From 048d345cd302d412565d6b214f1e912a788ffecb Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 02:58:56 +0000 Subject: [PATCH 01/19] Wire srv1 workspace image delivery and admitted allocation dispatch --- .github/workflows/fleet-converge.yml | 62 +++++++- dag/gunbc/ci/ci_spec.dag | 14 ++ dag/gunbc/fleet/fleet_converge_plan_cli.dag | 90 +++--------- dag/gunbc/fleet/fleet_converge_workflow.dag | 64 ++++++++- dag/gunbc/fleet/fleet_workflow_steps.dag | 13 +- dag/gunbc/live_deploy/apply.dag | 8 +- dag/gunbc/live_deploy/emit.dag | 30 +++- dag/gunbc/runner/workspace_guest_image.dag | 6 +- dag/gunbc/workspace_allocation_dispatch.dag | 132 ++++++++++++++++++ ...orkspace_allocation_dispatch_transport.dag | 51 +++++++ dag/gunbc/workspace_artifact_policy.dag | 2 +- .../workspace_commissioning_dispatch.dag | 6 + ...pace_commissioning_dispatch_operations.dag | 2 +- dag/gunbc/workspace_gateway_operator.dag | 76 ++++++++++ dag/gunbc/workspace_image_delivery.dag | 59 ++++++++ dag/gunbc/workspace_operator_request.dag | 79 ++++++++++- ...space_allocation_dispatch_witness_test.dag | 46 ++++++ ...orkspace_gateway_operator_witness_test.dag | 12 ++ ...orkspace_launch_directive_witness_test.dag | 6 +- ...rkspace_workflow_dispatch_witness_test.dag | 38 +++++ docs/plans/srv1-workspace-vm-bringup.md | 81 +++++++++++ 21 files changed, 783 insertions(+), 94 deletions(-) create mode 100644 dag/gunbc/workspace_allocation_dispatch.dag create mode 100644 dag/gunbc/workspace_allocation_dispatch_transport.dag create mode 100644 dag/gunbc/workspace_gateway_operator.dag create mode 100644 dag/gunbc/workspace_image_delivery.dag create mode 100644 dag/test/claim/workspace_allocation_dispatch_witness_test.dag create mode 100644 dag/test/claim/workspace_gateway_operator_witness_test.dag create mode 100644 dag/test/claim/workspace_workflow_dispatch_witness_test.dag create mode 100644 docs/plans/srv1-workspace-vm-bringup.md diff --git a/.github/workflows/fleet-converge.yml b/.github/workflows/fleet-converge.yml index 7900ff50b53..9384d957d26 100644 --- a/.github/workflows/fleet-converge.yml +++ b/.github/workflows/fleet-converge.yml @@ -5,6 +5,10 @@ run-name: fleet-converge ${{ inputs.mode }} ${{ inputs.host }} nonce=${{ inputs. on: workflow_dispatch: inputs: + workspace_image_run_id: + description: Image build run to download for controller install; installed bytes must match the reviewed image pin + required: false + type: string host: description: Executor fleet host (runner pinned here; plan mode observes this host) required: true @@ -13,7 +17,7 @@ on: mode: description: "plan shows membership hunks; apply executes a prior plan artifact; org_actions_observe validates the org credential and refuses on runner-group drift; org_runner_roster_observe reads the organization's self-hosted runner registrations to exhaustion under the same token and refuses unless the roster is complete; app_control_plane_observe reads the gunbai-ci App registration and webhook config with a short-lived App JWT and refuses on hook-config drift; app_key_version_verify reads the gunbai-ci App private key at the EXACT Secret Manager version named by app_key_version, mints an installation token with it, and refuses unless GitHub accepts it and the key's rotation deadline has not passed -- no add, disable or destroy; runner_browser_toolchain_converge installs the declared Playwright/Chromium toolchain (apt host libraries as the administrator, digest-pinned node, Playwright and Chromium archives into the job user's root) on the selected host, which must be in the pool that runs the floor job, and refuses unless every digest, version and host library reads back and headless Chromium renders a local page; microvm_host_converge installs the cited Firecracker release on the selected host and reads the kvm grant back, refusing by name when the grant has not landed (the grant itself is applied by the full-host apply spine); microvm_network_apply stages the slot and host network files the model renders at the named expected_revision as root:root 0600 in a root-only directory over the fleet SSH edge as the host's ADMINISTRATOR, installs them with the modeled operations, reloads networkd, systemd-sysctl and the nft loader unit, and reads the ruleset back -- the job user is granted none of it, because install plus systemctl over content that principal can write is arbitrary root for any pull request; guest_image_observe reports the micro-VM base artifacts and their measured digests on the selected host and builds nothing; guest_image_converge builds the runner guest image there; microvm_boot_probe boots that image and reads its serial console; spark_grants and spark_bootstrap prepare the selected Spark; spark_serving_apply applies the promoted fabric groups' pair vLLM units over the password session, workers before heads; spark_native_serving_apply relaunches group B's native four-rank arm as ONE BOUNDED TRANSACTION -- an all-host preflight that mutates nothing and refuses on any unresolved artifact from an earlier run, the incumbent UNIT preserved (plus a diagnostic inspect of the incumbent container -- the container itself cannot outlive its unit, because these run --rm in the foreground under systemd), the head applied BEFORE the workers so every rank joins the head's new rendezvous store, then a readback of the COMPLETE realization through each container's own inspect -- every field the create spec states, the effective --node-rank off that container's argv, and NCCL's typed transport verdict on that incarnation -- plus the head's front door answering with this arm's served model, and then commit or a rollback of the whole arm -- and it is a separate mode because it reloads a 328 GB arm and nobody should reach it by asking for the pair units; spark_runtime_image_probe pulls the pinned runtime image on the selected Spark and reads its capabilities from inside its own digest, changing nothing; spark_wireless_link_converge reads the selected Spark's Wi-Fi power save (runtime via iw, persisted via the NetworkManager profile), link and kernel disconnect count, sets whichever realization differs from the declared intent, and refuses unless the readback decides Noop; spark_grants_observe reads every procured Spark's sudo grant listing as gunbc-automation and the bootstrap principal, and its sudoers drop-in shape, and writes nothing; spark_v41_checkpoint_materialize fetches the admitted published DeepSeek V4.1 files onto the selected Group A Spark (about 510 GB; spark_v41_row_store_encode encodes the eight Engram row stores from the verified shards on the selected Group A Spark and reads each store's sha256; spark_v41_row_store_readback reads those stores back at their header, first and last record and every rank seam, with the published source rows at the same rows, and writes nothing; spark_v41_engram_differential compares upstream's Engram lookup kernel with the design-B file-backed lookup over sampled real rows of every row store, byte for byte, and writes nothing; it states the requirement and refuses before fetching when the disk cannot hold it), publishes each only after its sha256 matches the manifest, leaves a present file with the right digest alone and refuses one with the wrong digest, and reads the storage-backed Engram spans from the verified shards; a transfer runs detached and a rerun reattaches; spark_v41_runtime_image_build PRODUCES the DeepSeek V4.1 image on the selected Spark -- it verifies the candidate's three FlashInfer wheels against the digests the candidate keys, converges the patched source tree, builds from it, reads the produced configuration digest back from inside the image through the probe route, and admits that digest against the candidate's own recipe, refusing a digest that does not recompute from it -- and it is a separate mode from the probe because it occupies one host for hours where the probe occupies it for minutes; spark_v41_runtime_image_distribute moves that produced image, named by the configuration digest its production receipt read back, from the host that receipt names to the selected Group A Spark -- save into its fabric blob root, pulled by the target straight over the fabric rail, load -- after stating its size against every filesystem a copy lands on, leaves a target already holding the digest untouched, refuses a target holding a different image under the tag, and refuses unless the target's image inspect Id reads back as that digest; spark_v41_group_a_launch reads the production image back under its tag on every Group A host, reads its registry inside its digest and every host's occupancy, and only when Group A is suspended for this candidate with every host held and vacant stages the Engram manifest and applies the V4.1 four-rank arm as one transaction at the capacity measurement's shape -- the target names only the session host; spark_v41_serving_load runs the shared serving-load runner against the V4.1 head (target must be srv6): one vllm bench serve step per capacity-measurement concurrency, metrics scraped around each, host pressure read on every Group A rank, and the staircase stop rules applied over the worst rank; it changes no unit and writes only its receipt; host_reset_return drives a rostered reset subject through its controller and measures the return FROM A PEER, and takes reset_observer rather than host because the observer must not be the subject; runner_host_file_observe reads the runner teardown drop-in, the needrestart deferral and the loaded teardown on the selected host as its administrator and writes nothing; runner_host_file_converge writes whichever of those two files differs, reloads systemd only if the drop-in changed, and refuses unless the manager then reports the declared teardown; microvm_controller_install writes the microVM slot controller's root-owned release locus (gunbc + sources + Firecracker + jailer) and the gunbc-microvm-slot@ template unit and the gunbc-microvm-slot-reserve broker unit on srv1 and starts neither; microvm_slot_reserve starts that broker unit once on srv1, which reserves the shakedown cell through the fabric broker route from inside the root process (slot, demand and offer derived from the model, never inputs), and uploads that invocation's reservation receipt, failing unless the reservation committed; microvm_slot_start starts the shakedown slot's controller unit once on srv1 (the instance is derived from the model, never an input), waits for it bounded by the unit's own stop timeout, and uploads that invocation's controller receipt; microvm_runner_group_ensure (srv1 only) reads the organization's runner groups and, only when the microvm-shakedown group is absent, files ONE operator approval, creates it restricted to the shakedown workflow on the default branch, and refuses unless the readback holds that restriction; mtcollins1_census_qemu_host_observe reads the selected host's KVM device and, under the job user's census QEMU root, the qemu-system-aarch64 build, its ldd libraries and the AAVMF images against their pins, and writes nothing; mtcollins1_census_qemu_toolchain_converge places that root as the job user (digest-pinned noble .debs unpacked with dpkg-deb -x, no apt, no Recommends) and refuses unless the same observe reads it ready; workspace_source_pack (host=srv1) enumerates the operator workspace as the job user, drops every path the credential exclusion row names and every build output, tars exactly the remainder, reads the archive back and refuses if any member is excluded, and puts it into the workspace bucket under its SHA-256, refusing by name when the runner cannot read a source root; workspace_checkpoint_measure (host=srv3, workspace_source_object = that SHA-256) refuses unless the job user's home is on btrfs, then times a read-only subvolume snapshot, a pinned kopia scan of it, the content-addressed chunk upload and the gated revision commit, and receipts every time with the 10 s / 60 s / 10 GB draft verdicts; workspace_checkpoint_restore (host=srv3) commits a small authored workspace, puts one chunk with no head advance, destroys the directory, restores it from the head closure and requires byte-equality with the uncommitted chunk absent -- both srv3 modes refuse their commit while the R2 head's CAS ground is uncited" required: true - options: [plan, launch_environment_plan, allocation_store_plan, workspace_commissioning_plan, apply, org_actions_observe, app_control_plane_observe, microvm_host_converge, microvm_network_observe, microvm_network_apply, guest_image_observe, guest_image_converge, microvm_boot_probe, spark_grants, spark_bootstrap, spark_serving_apply, spark_native_serving_apply, spark_runtime_image_probe, spark_wireless_link_converge, spark_grants_observe, spark_v41_checkpoint_materialize, spark_v41_row_store_encode, spark_v41_row_store_readback, spark_v41_engram_differential, spark_runtime_image_produce, spark_runtime_image_distribute, spark_arm_group_load, spark_arm_checkpoint_materialize, spark_arm_group_observe, spark_arm_group_launch_plan, spark_arm_group_launch, spark_v41_serving_load, dashboard_deploy, approval_broker_dark_install, microvm_controller_install, microvm_slot_reserve, microvm_slot_start, rlm_launch_deployment_receipt, host_reset_return, runner_host_file_observe, runner_host_file_converge, site_pxe_edge_observe, site_pxe_edge_converge, runner_password_session_tool_converge, runner_browser_toolchain_converge, r2_mint_preflight, r2_object_write_mint, org_runner_roster_observe, approval_keyring_converge, approval_device_enrolment_code_issue, mtcollins1_boot, mtcollins1_fan_observe, mtcollins1_ui_bundle_observe, mtcollins1_kvm_observer_observe, mtcollins1_census_image_publish, mtcollins1_census_member_readback, mtcollins1_census_qemu_host_observe, mtcollins1_census_qemu_toolchain_converge, host_credential_custody_converge, app_key_version_verify, r2_bucket_ensure, r2_bucket_admin_mint, r2_workspace_object_read_mint, r2_workspace_object_write_mint, r2_cache_object_read_mint, r2_cache_object_write_mint, r2_conditional_put_race_probe, workspace_source_pack, workspace_checkpoint_measure, workspace_checkpoint_restore, fabric_writer_identity_observe, pair_serving_d0, microvm_runner_group_ensure, gcp_iam_converge, namecheap_observe, printer] + options: [plan, launch_environment_plan, allocation_store_plan, workspace_commissioning_plan, workspace_allocation_plan, workspace_image_build, apply, org_actions_observe, app_control_plane_observe, microvm_host_converge, microvm_network_observe, microvm_network_apply, guest_image_observe, guest_image_converge, microvm_boot_probe, spark_grants, spark_bootstrap, spark_serving_apply, spark_native_serving_apply, spark_runtime_image_probe, spark_wireless_link_converge, spark_grants_observe, spark_v41_checkpoint_materialize, spark_v41_row_store_encode, spark_v41_row_store_readback, spark_v41_engram_differential, spark_runtime_image_produce, spark_runtime_image_distribute, spark_arm_group_load, spark_arm_checkpoint_materialize, spark_arm_group_observe, spark_arm_group_launch_plan, spark_arm_group_launch, spark_v41_serving_load, dashboard_deploy, approval_broker_dark_install, microvm_controller_install, microvm_slot_reserve, microvm_slot_start, rlm_launch_deployment_receipt, host_reset_return, runner_host_file_observe, runner_host_file_converge, site_pxe_edge_observe, site_pxe_edge_converge, runner_password_session_tool_converge, runner_browser_toolchain_converge, r2_mint_preflight, r2_object_write_mint, org_runner_roster_observe, approval_keyring_converge, approval_device_enrolment_code_issue, mtcollins1_boot, mtcollins1_fan_observe, mtcollins1_ui_bundle_observe, mtcollins1_kvm_observer_observe, mtcollins1_census_image_publish, mtcollins1_census_member_readback, mtcollins1_census_qemu_host_observe, mtcollins1_census_qemu_toolchain_converge, host_credential_custody_converge, app_key_version_verify, r2_bucket_ensure, r2_bucket_admin_mint, r2_workspace_object_read_mint, r2_workspace_object_write_mint, r2_cache_object_read_mint, r2_cache_object_write_mint, r2_conditional_put_race_probe, workspace_source_pack, workspace_checkpoint_measure, workspace_checkpoint_restore, fabric_writer_identity_observe, pair_serving_d0, microvm_runner_group_ensure, gcp_iam_converge, namecheap_observe, printer] type: choice target: description: "Spark target host for the spark_* modes, naming the administrator credential the run materializes (any rostered Spark: srv5, srv6, srv7, srv8, srv9, srv10, srv11, srv12); apply reads the subject off the plan artifact" @@ -187,7 +191,7 @@ jobs: - name: Derive release-bins key (pre-materialization axes; sha256 at cross-run scope) id: release_bins_key run: | - if [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then + if [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_allocation_plan' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_image_build' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then set -e ROOT=$('git' 'rev-parse' '--show-toplevel') DIRTY=$('git' '-C' "$ROOT" 'status' '--porcelain' '--untracked-files=no') @@ -393,7 +397,7 @@ jobs: - name: Decide release-bins outcome; verify a stored pack completely before any restored binary runs id: release_bins_outcome run: | - if [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then + if [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_allocation_plan' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_image_build' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then set -e ROOT=$('git' 'rev-parse' '--show-toplevel') STAGE="$ROOT"'/.release-bins-staging' @@ -531,7 +535,7 @@ jobs: - name: Build release bins (the dispatched mode's product) id: release_build run: | - if [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then + if [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_allocation_plan' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_image_build' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then cargo build --release -p v1-compiler --bin claim_executor --bin gunbc else ROOT=$('git' 'rev-parse' '--show-toplevel') @@ -566,7 +570,7 @@ jobs: timeout-minutes: 5 - name: Pack release bins with member manifest into staging (build outcomes only; tar keeps exec bits) run: | - if [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then + if [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_allocation_plan' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_image_build' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then set -e ROOT=$('git' 'rev-parse' '--show-toplevel') STAGE="$ROOT"'/.release-bins-staging' @@ -639,7 +643,7 @@ jobs: - name: Verify the handed-off release bins against the build key, then promote and run --verify-build-artifacts (fail-closed) id: release_bins run: | - if [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then + if [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_allocation_plan' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_image_build' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then set -e ROOT=$('git' 'rev-parse' '--show-toplevel') '[' '-n' "$RELEASE_BINS_KEY" ']' || ('echo' '::error title=release-bins ConsumerKeyMissing::the build job published no release-bins key'; exit 1) @@ -776,6 +780,7 @@ jobs: service_account: fleet-cloud-convergence@gunbai-secrets.iam.gserviceaccount.com token_format: access_token create_credentials_file: false + if: "!(github.event.inputs.mode == 'workspace_image_build' || github.event.inputs.mode == 'workspace_allocation_plan')" timeout-minutes: 5 - name: Materialize fleet key in-run (SM versions/1 pinned -> RUNNER_TEMP 0600 -> ssh-agent -> wipe file) run: |- @@ -843,6 +848,40 @@ jobs: RLM_EXPECTED_REVISION: ${{ github.event.inputs.expected_revision }} if: github.event.inputs.mode == 'workspace_commissioning_plan' timeout-minutes: 15 + - name: "Workspace: workspace_allocation_plan" + id: workspace_allocation_plan + run: |- + ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) + "$ROOT/target/release/gunbc" run --source-root "$ROOT/dag" --source-root "$ROOT/src/v2" --entry dag/gunbc/fleet/fleet_converge_plan_cli.dag --function fleet_converge_workspace_allocation_plan_wet + env: + FLEET_CONVERGE_EXPECTED_HOST: ${{ inputs.host }} + RLM_EXPECTED_REVISION: ${{ inputs.expected_revision }} + if: github.event.inputs.mode == 'workspace_allocation_plan' + timeout-minutes: 30 + - name: "Workspace: workspace_image_build" + id: workspace_image_build + run: | + ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) + fakeroot -- "$ROOT/target/release/gunbc" run --source-root "$ROOT/dag" --source-root "$ROOT/src/v2" --entry dag/gunbc/workspace_image_delivery.dag --function workspace_image_build_wet + env: + FLEET_CONVERGE_EXPECTED_HOST: ${{ inputs.host }} + RLM_EXPECTED_REVISION: ${{ inputs.expected_revision }} + if: github.event.inputs.mode == 'workspace_image_build' + timeout-minutes: 30 + - name: Archive the exact workspace image build and digest receipt + id: workspace_image_upload + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f + with: + name: workspace-guest-image + path: |- + target/workspace-image-artifact/workspace.ext4 + target/workspace-image-artifact/base/vmlinux-6.1.128 + target/workspace-image-artifact/build-receipt.txt + if-no-files-found: error + retention-days: 90 + compression-level: 6 + if: github.event.inputs.mode == 'workspace_image_build' + timeout-minutes: 10 - name: Upload fleet converge plan artifact id: plan_upload uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f @@ -852,7 +891,7 @@ jobs: if-no-files-found: error retention-days: 30 compression-level: 0 - if: github.event.inputs.mode == 'plan' || github.event.inputs.mode == 'launch_environment_plan' || github.event.inputs.mode == 'allocation_store_plan' || github.event.inputs.mode == 'workspace_commissioning_plan' + if: github.event.inputs.mode == 'plan' || github.event.inputs.mode == 'launch_environment_plan' || github.event.inputs.mode == 'allocation_store_plan' || github.event.inputs.mode == 'workspace_commissioning_plan' || github.event.inputs.mode == 'workspace_allocation_plan' timeout-minutes: 10 - name: Download fleet converge plan artifact id: plan_download @@ -3178,6 +3217,15 @@ jobs: env: RELEASE_BINS_KEY: ${{ needs.build.outputs.release_bins_key }} timeout-minutes: 5 + - name: Download the workspace image selected for installation + id: workspace_image_download + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 + with: + name: workspace-guest-image + path: target/workspace-image-artifact + github-token: ${{ secrets.GITHUB_TOKEN }} + run-id: ${{ inputs.workspace_image_run_id }} + timeout-minutes: 10 - name: Install the microVM slot controller release locus, VMM + jailer and template unit on srv1 (additive; starts no instance) id: microvm_controller_install run: |- diff --git a/dag/gunbc/ci/ci_spec.dag b/dag/gunbc/ci/ci_spec.dag index d86d550ed34..577d7dd63df 100644 --- a/dag/gunbc/ci/ci_spec.dag +++ b/dag/gunbc/ci/ci_spec.dag @@ -1383,6 +1383,8 @@ fn gunbc_run_step_targets() -> List { gunbc_ci_fleet_converge_launch_environment_plan_target, gunbc_ci_fleet_converge_allocation_store_plan_target, gunbc_ci_fleet_converge_workspace_commissioning_plan_target, + gunbc_ci_workspace_image_build_target, + gunbc_ci_workspace_allocation_plan_target, gunbc_ci_fleet_converge_apply_target, gunbc_ci_org_actions_inspect_target, gunbc_ci_r2_mint_preflight_target, @@ -4069,3 +4071,15 @@ fn gunbc_ci_printer_workflow_invoke() -> String { function: gunbc_ci_printer_workflow_target.function, claim_run: false, receipt_rel: printer_workflow_receipt_path) } + +data gunbc_ci_workspace_allocation_plan_target: GunbcRunStepTarget = GunbcRunStepTarget { entry: "dag/gunbc/fleet/fleet_converge_plan_cli.dag", function: "fleet_converge_workspace_allocation_plan_wet" } + +fn gunbc_ci_workspace_allocation_plan_invoke() -> String { + gunbc_run_step_script(source_roots: witness_layer_roots, entry: gunbc_ci_workspace_allocation_plan_target.entry, function: gunbc_ci_workspace_allocation_plan_target.function, claim_run: false, receipt_rel: none) +} + +data gunbc_ci_workspace_image_build_target: GunbcRunStepTarget = GunbcRunStepTarget { entry: "dag/gunbc/workspace_image_delivery.dag", function: "workspace_image_build_wet" } + +fn gunbc_ci_workspace_image_build_invoke() -> String { + join([gunbc_invoke_root_stamp_script(), "fakeroot -- ", gunbc_run_command(source_roots: witness_layer_roots, entry: gunbc_ci_workspace_image_build_target.entry, function: gunbc_ci_workspace_image_build_target.function, claim_run: false), "\n"], "") +} diff --git a/dag/gunbc/fleet/fleet_converge_plan_cli.dag b/dag/gunbc/fleet/fleet_converge_plan_cli.dag index 1725ef7d31c..e9df9ddd8d6 100644 --- a/dag/gunbc/fleet/fleet_converge_plan_cli.dag +++ b/dag/gunbc/fleet/fleet_converge_plan_cli.dag @@ -1,5 +1,8 @@ module gunbc.fleet_converge_plan_cli +import gunbc.live_deploy.release_locus { approval_broker_release_dir, RevisionBoundAtEmission } +import gunbc.runner_microvm_slot_unit { microvm_controller_root } + import gunbc.fleet_converge_scope { WorkspaceAllocationOnly, WorkspaceSlotCommissioningOnly, FleetConvergeScope, FullHost, FabricExecutionCellsOnly, FabricAllocationStoreOnly, LiveDeploy, LaunchEnvironment } import gunbc.fleet_plan_artifact_location { fleet_converge_plan_artifact_dir } @@ -15,13 +18,12 @@ import std.types { CommitSha } import gunbc.fabric_storage_binding { fabric_state_storage_binding } -import gunbc.workspace_allocation_convergence { workspace_convergence_observe, WorkspaceConvergenceObserved, WorkspaceConvergenceUnobserved } +import gunbc.workspace_allocation_convergence { WorkspaceConvergenceObserved, WorkspaceConvergenceUnobserved } import gunbc.workspace_allocation_launch_directive { workspace_launch_directive_from_text, WorkspaceLaunchDirectiveReady, WorkspaceLaunchDirectiveRefused } -import gunbc.workspace_reconcile_once { workspace_reconcile_prepare } -import gunbc.workspace_allocation_store { WorkspaceAllocation, workspace_allocation_json, workspace_record_preparation_wait, WorkspaceStoreWritten, WorkspaceStoreConflict, WorkspaceStoreWriteRefused } -import gunbc.workspace_request_consumer { workspace_consumer_reconcile_enrolled, WorkspaceConsumerIdle, WorkspaceConsumerRefused, WorkspaceConsumerRetire, WorkspaceConsumerPrepare, WorkspaceConsumerRoute, WorkspaceConsumerSettled, WorkspaceConsumerRetry } +import gunbc.workspace_allocation_dispatch_transport { workspace_allocation_dispatch_prepare, workspace_allocation_dispatch_readback } +import gunbc.workspace_allocation_store { workspace_allocation_json } import gunbc.workspace_allocation_policy { workspace_installed_policy } -import gunbc.workspace_allocation_admission { workspace_owner_enrollment, workspace_enrolled_owners, WorkspacePolicyAdmitted, WorkspacePolicyRefused } +import gunbc.workspace_allocation_admission { workspace_owner_enrollment, WorkspacePolicyAdmitted, WorkspacePolicyRefused } import gunbc.principal_projection { PrincipalRef } import extdeps.languages.json.emit { serialize_json } @@ -413,7 +415,7 @@ fn observe_fleet_converge_request_wet(scope_wire: String, host: NonEmptyStr) -> }, } ScopeDecoded { scope: WorkspaceSlotCommissioningOnly } => FleetConvergeRequestUnobserved { reason: "commissioning observation requires the admitted revision" } - ScopeDecoded { scope: WorkspaceAllocationOnly } => observe_workspace_allocation_plan_request_wet(host: host) + ScopeDecoded { scope: WorkspaceAllocationOnly } => FleetConvergeRequestUnobserved { reason: "workspace allocation observation requires the admitted revision" } ScopeDecoded { scope: LiveDeploy } => FleetConvergeRequestUnobserved { reason: "observe_fleet_converge_request_wet is host-and-scope-wire; LiveDeployConverge is minted from DesiredDeployment and LiveDeployMemberBasis, not a hostname", @@ -880,6 +882,10 @@ fn fleet_converge_plan_scoped_bound_wet( WorkspaceCommissioningFleetObserved { basis } => FleetConvergeRequestObserved { request: WorkspaceSlotCommissioningConverge { basis: basis } } WorkspaceCommissioningFleetRefused { reason: _ } => FleetConvergeRequestUnobserved { reason: "commissioning privileged observation refused" } } + } else if (scope == WorkspaceAllocationOnly) { + observe_workspace_allocation_plan_request_wet(host: subject_host, + release_dir: approval_broker_release_dir(root: microvm_controller_root, + revision: RevisionBoundAtEmission { revision: run.run_revision as CommitSha }) as NonEmptyStr) } else { observe_fleet_converge_request_wet(scope_wire: fleet_converge_scope_wire(scope: scope), host: subject_host) } match observed { FleetConvergeRequestUnobserved { reason: why } => @@ -1586,10 +1592,7 @@ fn observe_fleet_converge_request_from_plan_wet(scope_wire: String, host: NonEmp WorkspaceLaunchDirectiveRefused { reason: detail } => FleetConvergeRequestUnobserved { reason: detail } WorkspaceLaunchDirectiveReady { directive: planned } => if !(match workspace_installed_policy() { WorkspacePolicyAdmitted { policy } => (match workspace_owner_enrollment(policy: policy, owner: planned.allocation.request.owner) { Present { value: _ } => true Absent => false }) WorkspacePolicyRefused { reason: _ } => false }) { FleetConvergeRequestUnobserved { reason: "workspace plan owner is not authorized by the installed allocation policy" } - } else { match workspace_convergence_observe( - store: fabric_state_storage_binding(), - owner: planned.allocation.request.owner, idempotency_key: planned.allocation.request.idempotency_key, - ) { + } else { match workspace_allocation_dispatch_readback(host: host, planned: planned) { WorkspaceConvergenceUnobserved { reason: detail } => FleetConvergeRequestUnobserved { reason: detail } WorkspaceConvergenceObserved { basis: current } => if current.owner_generation != planned.owner_generation || serialize_json(v: workspace_allocation_json(allocation: current.allocation)) != serialize_json(v: workspace_allocation_json(allocation: planned.allocation)) { @@ -1603,64 +1606,11 @@ fn observe_fleet_converge_request_from_plan_wet(scope_wire: String, host: NonEmp } } -// Discover the outstanding request from protected state rather than requiring an HTTP process -// to remember or forward its identity. Retirement runs before installed-artifact or offer reads. -fn observe_workspace_allocation_plan_request_wet(host: NonEmptyStr) -> FleetConvergeRequestObservation { - match clock_now_probed_at() { - Absent => FleetConvergeRequestUnobserved { reason: "workspace consumer clock is unavailable" } - Present { value: now } => match workspace_installed_policy() { - WorkspacePolicyRefused { reason: detail } => FleetConvergeRequestUnobserved { reason: detail } - WorkspacePolicyAdmitted { policy } => match workspace_consumer_reconcile_enrolled(store: fabric_state_storage_binding(), - owners: workspace_enrolled_owners(policy: policy), observed_at: now as String) { - WorkspaceConsumerIdle => FleetConvergeRequestUnobserved { reason: "no outstanding workspace request" } - WorkspaceConsumerRefused { reason: detail } => FleetConvergeRequestUnobserved { reason: detail } - WorkspaceConsumerRetry => FleetConvergeRequestUnobserved { reason: "workspace owner changed concurrently; reconcile again" } - WorkspaceConsumerRetire { allocation: _ } => FleetConvergeRequestUnobserved { reason: "workspace retirement remains outstanding" } - WorkspaceConsumerSettled { allocation: _ } => FleetConvergeRequestUnobserved { reason: "unplanned workspace retirement settled; no host apply required" } - WorkspaceConsumerPrepare { allocation: allocation } => match shell.Env.Get(name: "GUNBC_WORKSPACE_CONTROLLER_RELEASE" as NonEmptyStr).value { - Absent => workspace_publish_preparation_wait(allocation: allocation, reason: "workspace preparation requires the installed controller release identity") - Present { value: installed } => if trim(s: installed) == "" { - workspace_publish_preparation_wait(allocation: allocation, reason: "workspace installed controller release identity is empty") - } else { match workspace_reconcile_prepare(store: fabric_state_storage_binding(), allocation: allocation, - local_host: host as HostIdentity, release_dir: trim(s: installed) as NonEmptyStr) { - WorkspaceConvergenceUnobserved { reason: detail } => workspace_publish_preparation_wait(allocation: allocation, reason: detail) - WorkspaceConvergenceObserved { basis: planned } => if (planned.host as String) != (host as String) { - FleetConvergeRequestUnobserved { reason: concat("workspace plan selected host ", concat(planned.host as String, "; dispatch fleet-converge allocation plan there")) } - } else { observe_workspace_prepared_plan_request_wet(owner: allocation.request.owner, request_key: allocation.request.idempotency_key, host: host) } - } } - } - WorkspaceConsumerRoute { allocation: allocation } => observe_workspace_prepared_plan_request_wet( - owner: allocation.request.owner, request_key: allocation.request.idempotency_key, host: host) - } - } - } -} - -// A preparation that could not select publishes its located reason for the owner's view and stops. -fn workspace_publish_preparation_wait(allocation: WorkspaceAllocation, reason: String) -> FleetConvergeRequestObservation { - match workspace_record_preparation_wait(store: fabric_state_storage_binding(), request: allocation.request, reason: reason) { - WorkspaceStoreWritten { allocation: _ } => FleetConvergeRequestUnobserved { reason: reason } - WorkspaceStoreConflict => FleetConvergeRequestUnobserved { reason: "workspace changed while publishing preparation wait; reconcile again" } - WorkspaceStoreWriteRefused { reason: detail } => FleetConvergeRequestUnobserved { reason: detail } - } -} - -// Local authorities are read only after the existing selected-host check. Apply still obtains -// its identity and installed artifact from the hash-admitted plan, never from this environment. -fn observe_workspace_prepared_plan_request_wet(owner: PrincipalRef, request_key: NonEmptyStr, host: NonEmptyStr) -> FleetConvergeRequestObservation { - match workspace_convergence_observe(store: fabric_state_storage_binding(), - owner: owner, idempotency_key: request_key) { - WorkspaceConvergenceUnobserved { reason: detail } => FleetConvergeRequestUnobserved { reason: detail } - WorkspaceConvergenceObserved { basis: selected } => if selected.host != host { - FleetConvergeRequestUnobserved { reason: "workspace plan request is executing on a different host" } - } else { match shell.Env.Get(name: "GUNBC_WORKSPACE_CONTROLLER_RELEASE" as NonEmptyStr).value { - Absent => FleetConvergeRequestUnobserved { reason: "workspace plan requires the installed controller release identity" } - Present { value: installed } => if trim(s: installed) == "" { - FleetConvergeRequestUnobserved { reason: "workspace installed controller release identity is empty" } - } else { FleetConvergeRequestObserved { - request: WorkspaceAllocationConverge { basis: selected, release_dir: trim(s: installed) as NonEmptyStr }, - } } - } } +// State access uses the exact root-owned installed helper. The runner never receives the MAC key. +fn observe_workspace_allocation_plan_request_wet(host: NonEmptyStr, release_dir: NonEmptyStr) -> FleetConvergeRequestObservation { + match workspace_allocation_dispatch_prepare(host: host, release_dir: release_dir) { + WorkspaceConvergenceUnobserved { reason } => FleetConvergeRequestUnobserved { reason: reason } + WorkspaceConvergenceObserved { basis } => FleetConvergeRequestObserved { request: WorkspaceAllocationConverge { basis: basis, release_dir: release_dir } } } } @@ -1759,3 +1709,7 @@ fn fleet_converge_apply_commissioning_bound_wet(run: ActionsRunBinding, plan_run fn fleet_converge_workspace_commissioning_plan_wet() -> ProcessExit { fleet_converge_scoped_plan_wet(scope: WorkspaceSlotCommissioningOnly) } + +fn fleet_converge_workspace_allocation_plan_wet() -> ProcessExit { + fleet_converge_scoped_plan_wet(scope: WorkspaceAllocationOnly) +} diff --git a/dag/gunbc/fleet/fleet_converge_workflow.dag b/dag/gunbc/fleet/fleet_converge_workflow.dag index a5898bdb942..9c3c09e4d59 100644 --- a/dag/gunbc/fleet/fleet_converge_workflow.dag +++ b/dag/gunbc/fleet/fleet_converge_workflow.dag @@ -1,5 +1,8 @@ module gunbc.fleet_converge_workflow +import gunbc.ci_spec { gunbc_ci_workspace_allocation_plan_invoke, gunbc_ci_workspace_image_build_invoke } +import gunbc.workspace_image_delivery { workspace_image_artifact_name, workspace_image_staging_root, workspace_image_artifact_paths } + import gunbc.fleet_printer_access {printer_access_code_loci} import gunbc.auth.approval_keyring_pins {approval_submission_mac_key_path_env,approval_capability_mac_key_path_env,approval_store_receipt_mac_key_path_env} import gunbc.auth.approval_decision_store {approval_mac_key_path} @@ -8,7 +11,7 @@ import gunbc.auth.approval_store_receipt {approval_store_receipt_mac_key_path} import gunbc.fleet.printer_workflow_artifact { printer_workflow_receipt_path } import extdeps.github.expressions { Expression, ContextAccess, Github, StringLiteral, Equals, LogicalOr, LogicalAnd, interpolate } -import gunbc.fleet_converge_scope { FleetConvergeScope, FullHost, LaunchEnvironment, FabricAllocationStoreOnly, WorkspaceSlotCommissioningOnly, LiveDeploy } +import gunbc.fleet_converge_scope { FleetConvergeScope, FullHost, LaunchEnvironment, FabricAllocationStoreOnly, WorkspaceSlotCommissioningOnly, WorkspaceAllocationOnly, LiveDeploy } import gunbc.fleet_plan_artifact_location { fleet_converge_plan_artifact_dir } import gunbc.namecheap.observation_artifact { namecheap_observation_receipt_path } @@ -94,7 +97,7 @@ import gunbc.fleet_workflow_steps { ci_release_bins_outcome_step_for, ci_release_build_step_for, ci_release_bins_pack_step_for, - ci_fleet_wif_auth_step, + ci_fleet_wif_auth_step, ci_fleet_wif_auth_step_when, ci_fleet_key_agent_step, ci_fleet_key_agent_step_when, ci_fleet_key_agent_cleanup_step, @@ -308,6 +311,8 @@ type FleetConvergeWorkflowMode | LaunchEnvironmentPlan | AllocationStorePlan | WorkspaceCommissioningPlan + | WorkspaceAllocationPlan + | WorkspaceImageBuild | FleetApply | OrgActionsObserve | AppControlPlaneObserve @@ -387,6 +392,8 @@ fn fleet_converge_workflow_mode_wire(mode: FleetConvergeWorkflowMode) -> String LaunchEnvironmentPlan => "launch_environment_plan" AllocationStorePlan => "allocation_store_plan" WorkspaceCommissioningPlan => "workspace_commissioning_plan" + WorkspaceAllocationPlan => "workspace_allocation_plan" + WorkspaceImageBuild => "workspace_image_build" FleetApply => "apply" OrgActionsObserve => "org_actions_observe" AppControlPlaneObserve => "app_control_plane_observe" @@ -483,7 +490,7 @@ fn fleet_converge_spark_target_description() -> String { ], "") } -data fleet_converge_workflow_modes: List = [FullHostPlan, LaunchEnvironmentPlan, AllocationStorePlan, WorkspaceCommissioningPlan, FleetApply, OrgActionsObserve, AppControlPlaneObserve, MicrovmHostConverge, MicrovmNetworkObserve, MicrovmNetworkApply, GuestImageObserve, GuestImageConverge, MicrovmBootProbe, SparkGrants, SparkBootstrap, SparkServingApply, SparkNativeServingApply, SparkRuntimeImageProbe, SparkWirelessLinkConverge, SparkGrantsObserve, SparkV41CheckpointMaterialize, SparkV41RowStoreEncode, SparkV41RowStoreReadback, SparkV41EngramDifferential, SparkRuntimeImageProduce, SparkRuntimeImageDistribute, SparkArmGroupLoad, SparkArmCheckpointMaterialize, SparkArmGroupObserve, SparkArmGroupLaunchPlan, SparkArmGroupLaunch, SparkV41ServingLoad, DashboardDeploy, ApprovalBrokerDarkInstall, MicrovmControllerInstall, MicrovmSlotReserve, MicrovmSlotStart, RlmLaunchDeploymentReceipt, HostResetReturn, RunnerHostFileObserve, RunnerHostFileConverge, SitePxeEdgeObserve, SitePxeEdgeConverge, RunnerPasswordSessionToolConverge, RunnerBrowserToolchainConverge, R2MintPreflight, R2ObjectWriteMint, OrgRunnerRosterObserve, ApprovalKeyringConverge, ApprovalDeviceEnrolmentCodeIssue, MtCollins1Boot, MtCollins1FanObserve, MtCollins1UiBundleObserve, MtCollins1KvmObserverObserve, MtCollins1CensusImagePublish, MtCollins1CensusMemberReadback, MtCollins1CensusQemuHostObserve, MtCollins1CensusQemuToolchainConverge, HostCredentialCustodyConverge, AppKeyVersionVerify, R2BucketEnsure, R2BucketAdminMint, R2WorkspaceObjectReadMint, R2WorkspaceObjectWriteMint, R2CacheObjectReadMint, R2CacheObjectWriteMint, R2ConditionalPutRaceProbe, WorkspaceSourcePack, WorkspaceCheckpointMeasure, WorkspaceCheckpointRestore, FabricWriterIdentityObserve, PairServingD0, MicrovmRunnerGroupEnsure, GcpIamConverge, NamecheapObserve, PrinterWorkflow] +data fleet_converge_workflow_modes: List = [FullHostPlan, LaunchEnvironmentPlan, AllocationStorePlan, WorkspaceCommissioningPlan, WorkspaceAllocationPlan, WorkspaceImageBuild, FleetApply, OrgActionsObserve, AppControlPlaneObserve, MicrovmHostConverge, MicrovmNetworkObserve, MicrovmNetworkApply, GuestImageObserve, GuestImageConverge, MicrovmBootProbe, SparkGrants, SparkBootstrap, SparkServingApply, SparkNativeServingApply, SparkRuntimeImageProbe, SparkWirelessLinkConverge, SparkGrantsObserve, SparkV41CheckpointMaterialize, SparkV41RowStoreEncode, SparkV41RowStoreReadback, SparkV41EngramDifferential, SparkRuntimeImageProduce, SparkRuntimeImageDistribute, SparkArmGroupLoad, SparkArmCheckpointMaterialize, SparkArmGroupObserve, SparkArmGroupLaunchPlan, SparkArmGroupLaunch, SparkV41ServingLoad, DashboardDeploy, ApprovalBrokerDarkInstall, MicrovmControllerInstall, MicrovmSlotReserve, MicrovmSlotStart, RlmLaunchDeploymentReceipt, HostResetReturn, RunnerHostFileObserve, RunnerHostFileConverge, SitePxeEdgeObserve, SitePxeEdgeConverge, RunnerPasswordSessionToolConverge, RunnerBrowserToolchainConverge, R2MintPreflight, R2ObjectWriteMint, OrgRunnerRosterObserve, ApprovalKeyringConverge, ApprovalDeviceEnrolmentCodeIssue, MtCollins1Boot, MtCollins1FanObserve, MtCollins1UiBundleObserve, MtCollins1KvmObserverObserve, MtCollins1CensusImagePublish, MtCollins1CensusMemberReadback, MtCollins1CensusQemuHostObserve, MtCollins1CensusQemuToolchainConverge, HostCredentialCustodyConverge, AppKeyVersionVerify, R2BucketEnsure, R2BucketAdminMint, R2WorkspaceObjectReadMint, R2WorkspaceObjectWriteMint, R2CacheObjectReadMint, R2CacheObjectWriteMint, R2ConditionalPutRaceProbe, WorkspaceSourcePack, WorkspaceCheckpointMeasure, WorkspaceCheckpointRestore, FabricWriterIdentityObserve, PairServingD0, MicrovmRunnerGroupEnsure, GcpIamConverge, NamecheapObserve, PrinterWorkflow] // WHICH RELEASE PRODUCT A MODE'S JOB RUNS, NAMED PER MODE WITH NO WILDCARD (as the scope and fleet-key // matches above name theirs). A mode whose steps run only gunbc and claim_executor declares the // compiler pair (gunbc.fleet_release_bins_key compiler_pair_release_product): its build job then @@ -498,6 +505,8 @@ fn fleet_converge_mode_release_product(mode: FleetConvergeWorkflowMode) -> Fleet LaunchEnvironmentPlan => SixteenBinaryPack AllocationStorePlan => SixteenBinaryPack WorkspaceCommissioningPlan => SixteenBinaryPack + WorkspaceAllocationPlan => CompilerPairOnly + WorkspaceImageBuild => CompilerPairOnly FleetApply => SixteenBinaryPack OrgActionsObserve => SixteenBinaryPack AppControlPlaneObserve => SixteenBinaryPack @@ -609,6 +618,8 @@ fn fleet_converge_mode_scope(mode: FleetConvergeWorkflowMode) -> FleetConvergeSc LaunchEnvironmentPlan => Present { value: LaunchEnvironment } AllocationStorePlan => Present { value: FabricAllocationStoreOnly } WorkspaceCommissioningPlan => Present { value: WorkspaceSlotCommissioningOnly } + WorkspaceAllocationPlan => Present { value: WorkspaceAllocationOnly } + WorkspaceImageBuild => none FleetApply => none OrgActionsObserve => none AppControlPlaneObserve => none @@ -772,6 +783,8 @@ fn fleet_converge_mode_fleet_ssh_key_demand(mode: FleetConvergeWorkflowMode) -> LaunchEnvironmentPlan => FleetSshKeyConsumed AllocationStorePlan => FleetSshKeyConsumed WorkspaceCommissioningPlan => FleetSshKeyConsumed + WorkspaceAllocationPlan => FleetSshKeyNotConsumed + WorkspaceImageBuild => FleetSshKeyNotConsumed FleetApply => FleetSshKeyConsumed AppControlPlaneObserve => FleetSshKeyConsumed MicrovmHostConverge => FleetSshKeyConsumed @@ -1231,7 +1244,7 @@ data fleet_converge_launch_environment_plan_step_if: String = fleet_converge_mod data fleet_converge_allocation_store_plan_step_if: String = fleet_converge_mode_step_if(mode: AllocationStorePlan) data fleet_converge_workspace_commissioning_plan_step_if: String = fleet_converge_mode_step_if(mode: WorkspaceCommissioningPlan) -data fleet_converge_any_plan_step_if: String = fleet_converge_any_mode_step_if(modes: [FullHostPlan, LaunchEnvironmentPlan, AllocationStorePlan, WorkspaceCommissioningPlan]) +data fleet_converge_any_plan_step_if: String = fleet_converge_any_mode_step_if(modes: [FullHostPlan, LaunchEnvironmentPlan, AllocationStorePlan, WorkspaceCommissioningPlan, WorkspaceAllocationPlan]) data fleet_converge_apply_step_if: String = fleet_converge_mode_step_if(mode: FleetApply) data fleet_converge_org_actions_step_if: String = fleet_converge_mode_step_if(mode: OrgActionsObserve) data fleet_converge_org_runner_roster_step_if: String = fleet_converge_mode_step_if(mode: OrgRunnerRosterObserve) @@ -1435,6 +1448,8 @@ fn fleet_converge_mode_mutation_domain(mode: FleetConvergeWorkflowMode) -> Fleet LaunchEnvironmentPlan => ExecutorDomain AllocationStorePlan => ExecutorDomain WorkspaceCommissioningPlan => ExecutorDomain + WorkspaceAllocationPlan => ExecutorDomain + WorkspaceImageBuild => ExecutorDomain FleetApply => ExecutorDomain OrgActionsObserve => ExecutorDomain AppControlPlaneObserve => ExecutorDomain @@ -3633,6 +3648,9 @@ fn fleet_converge_microvm_controller_install_job() -> Job { [ ci_release_bins_download_step(), ci_release_bins_unpack_verify_step(), + fleet_converge_receipt_download_step(id: "workspace_image_download", name: "Download the workspace image selected for installation", + artifact: workspace_image_artifact_name, path: workspace_image_staging_root, + run_id_expression: "${{ inputs.workspace_image_run_id }}"), fleet_converge_microvm_controller_install_step(), ] ), @@ -4426,7 +4444,7 @@ fn fleet_converge_credential_prelude_steps() -> List { ci_release_bins_download_step(), ci_release_bins_unpack_verify_step_for(pair_modes: fleet_converge_compiler_pair_mode_wires()), fleet_converge_pair_serving_d0_admit_step(), - ci_fleet_wif_auth_step(), + ci_fleet_wif_auth_step_when(if_condition: Present { value: concat("!(", concat(fleet_converge_any_mode_step_if(modes: [WorkspaceImageBuild, WorkspaceAllocationPlan]), ")")) }), ci_fleet_key_agent_step_when(if_condition: Present { value: fleet_converge_fleet_ssh_key_step_if }), ] } @@ -4447,6 +4465,9 @@ fn fleet_converge_job() -> Job { fleet_converge_launch_environment_plan_step(), fleet_converge_allocation_store_plan_step(), fleet_converge_workspace_commissioning_plan_step(), + fleet_converge_workspace_operation_step(mode: WorkspaceAllocationPlan, run: gunbc_ci_workspace_allocation_plan_invoke()), + fleet_converge_workspace_operation_step(mode: WorkspaceImageBuild, run: gunbc_ci_workspace_image_build_invoke()), + fleet_converge_workspace_image_upload_step(), fleet_converge_plan_upload_step(), fleet_converge_plan_download_step(), fleet_converge_apply_step(), @@ -4698,6 +4719,8 @@ fn fleet_converge_mode_job_id(mode: FleetConvergeWorkflowMode) -> NonEmptyStr { LaunchEnvironmentPlan => fleet_converge_shared_job_id AllocationStorePlan => fleet_converge_shared_job_id WorkspaceCommissioningPlan => fleet_converge_shared_job_id + WorkspaceAllocationPlan => fleet_converge_shared_job_id + WorkspaceImageBuild => fleet_converge_shared_job_id FleetApply => fleet_converge_shared_job_id OrgActionsObserve => fleet_converge_shared_job_id AppControlPlaneObserve => fleet_converge_shared_job_id @@ -4976,6 +4999,8 @@ fn fleet_converge_gcp_iam_converge_job() -> Job { // THE DISPATCH INPUTS, AS THEIR OWN ROW: the workflow consumes them below, and a witness over // what the operator can name reads this list rather than building every job to reach it. data fleet_converge_dispatch_inputs: List = [ + DispatchInput { name: "workspace_image_run_id", description: Present { value: "Image build run to download for controller install; installed bytes must match the reviewed image pin" }, required: false, default: none, type: InputString }, + DispatchInput { name: "host", description: Present { value: "Executor fleet host (runner pinned here; plan mode observes this host)" }, @@ -5346,3 +5371,32 @@ fn fleet_converge_printer_job() -> Job { } }, } } + +// Image production and allocation preparation share the bounded host job; allocation uses reviewed apply. +fn fleet_converge_workspace_operation_step(mode: FleetConvergeWorkflowMode, run: String) -> Step { + RunStep { + name: Present { value: concat("Workspace: ", fleet_converge_workflow_mode_wire(mode: mode)) }, + id: Present { value: fleet_converge_workflow_mode_wire(mode: mode) }, run: run, shell: none, + env: Present { value: [ + kv(key: fleet_converge_expected_host_env_name, value: yaml_string(s: "${{ inputs.host }}")), + kv(key: rlm_expected_revision_env_name, value: yaml_string(s: "${{ inputs.expected_revision }}")), + ] }, + working_directory: none, if_condition: Present { value: fleet_converge_mode_step_if(mode: mode) }, + continue_on_error: none, timeout_minutes: Present { value: gunbc_ci_runner_guest_image_step_timeout_minutes } + } +} + +fn fleet_converge_workspace_image_upload_step() -> Step { + UsesStep { + name: Present { value: "Archive the exact workspace image build and digest receipt" }, + id: Present { value: "workspace_image_upload" }, uses: upload_artifact_action, + with: Present { value: [ + kv(key: "name", value: yaml_string(s: workspace_image_artifact_name)), + kv(key: "path", value: yaml_string(s: workspace_image_artifact_paths())), + kv(key: "if-no-files-found", value: yaml_string(s: "error")), + kv(key: "retention-days", value: yaml_int(n: 90)), + kv(key: "compression-level", value: yaml_int(n: 6)), + ] }, env: none, if_condition: Present { value: fleet_converge_mode_step_if(mode: WorkspaceImageBuild) }, + continue_on_error: none, timeout_minutes: Present { value: gunbc_ci_artifact_transfer_step_timeout_minutes } + } +} diff --git a/dag/gunbc/fleet/fleet_workflow_steps.dag b/dag/gunbc/fleet/fleet_workflow_steps.dag index 63a02d8cf08..c6fab534de5 100644 --- a/dag/gunbc/fleet/fleet_workflow_steps.dag +++ b/dag/gunbc/fleet/fleet_workflow_steps.dag @@ -562,9 +562,12 @@ fn ci_release_bins_unpack_verify_step() -> Step { } } -fn ci_fleet_wif_auth_step() -> Step { - ci_wif_auth_step_for( +fn ci_fleet_wif_auth_step() -> Step { ci_fleet_wif_auth_step_when(if_condition: none) } + +fn ci_fleet_wif_auth_step_when(if_condition: String?) -> Step { + ci_wif_auth_step_with_condition( name: "WIF auth (OIDC -> fleet-cloud-convergence SA, access token only)", + id: "wif_auth", if_condition: if_condition, provider_resource: probe_identity_secret_access_requirement.wif_provider_resource, service_account: probe_identity_secret_access_requirement.service_account, ) @@ -580,6 +583,10 @@ fn ci_wif_auth_step_for(name: String, provider_resource: String, service_account // The same step under a caller-chosen id, for a job that federates to more than one identity under // disjoint conditions (step ids are unique per job even when their conditions are exclusive). fn ci_wif_auth_step_with_id(name: String, id: String, provider_resource: String, service_account: String) -> Step { + ci_wif_auth_step_with_condition(name: name, id: id, provider_resource: provider_resource, service_account: service_account, if_condition: none) +} + +fn ci_wif_auth_step_with_condition(name: String, id: String, provider_resource: String, service_account: String, if_condition: String?) -> Step { UsesStep { name: Present { value: name }, id: Present { value: id }, @@ -593,7 +600,7 @@ fn ci_wif_auth_step_with_id(name: String, id: String, provider_resource: String, ] }, env: none, - if_condition: none, + if_condition: if_condition, continue_on_error: none, timeout_minutes: Present { value: gunbc_ci_aux_step_timeout_minutes } } diff --git a/dag/gunbc/live_deploy/apply.dag b/dag/gunbc/live_deploy/apply.dag index b6e8cd72b3f..0abd11d7880 100644 --- a/dag/gunbc/live_deploy/apply.dag +++ b/dag/gunbc/live_deploy/apply.dag @@ -1,5 +1,8 @@ module gunbc.live_deploy.apply +import gunbc.workspace_artifact_policy { workspace_demo_image_identity } +import gunbc.workspace_guest_image { workspace_installed_image, WorkspaceImageVerified, WorkspaceImageRefused } + import gunbc.workspace_commissioning_executor_input { workspace_executor_installed_adapter_matches } import gunbc.workspace_commissioning_executor_unit { workspace_executor_unit_instance_name } import gunbc.runner_microvm_slot_unit { microvm_controller_requires_ci_reservation_unit } @@ -1052,7 +1055,10 @@ fn microvm_controller_install_readback_exit(candidate: CandidateRelease) -> Proc ExitSuccess => if workspace_executor_installed_adapter_matches( release_dir: approval_broker_release_dir(root: microvm_controller_root, revision: revision), slot: runner_slot_instance_name(slot: srv1_microvm_shakedown_slot())) { - ExitSuccess + match workspace_installed_image(release_dir: approval_broker_release_dir(root: microvm_controller_root, revision: revision), identity: workspace_demo_image_identity()) { + WorkspaceImageVerified { image: _ } => ExitSuccess + WorkspaceImageRefused { reason } => exit_failure(reason: concat("microvm_controller_install: ", reason)) + } } else { exit_failure(reason: "microvm_controller_install: executor scripts or loaded unit failed exact readback") } refused => refused } diff --git a/dag/gunbc/live_deploy/emit.dag b/dag/gunbc/live_deploy/emit.dag index ddf5413f7a0..008b711c3d4 100644 --- a/dag/gunbc/live_deploy/emit.dag +++ b/dag/gunbc/live_deploy/emit.dag @@ -1,5 +1,10 @@ module gunbc.live_deploy.emit +import gunbc.workspace_artifact_policy { workspace_demo_image_identity } +import gunbc.workspace_image_delivery { workspace_image_staging_root } +import gunbc.workspace_guest_image { workspace_image_path, workspace_installed_image_root } +import gunbc.microvm_guest_artifacts { runner_guest_kernel_artifact, runner_guest_base_artifact_path, ArtifactDigestPinned, ArtifactDigestUnpinned, ArtifactDigestNotPinnable } + import gunbc.workspace_commissioning_dispatch_operations { workspace_dispatch_grant_path, workspace_dispatch_grant_content } import gunbc.live_deploy.release_environment { installed_release_environment, installed_release_unit } @@ -2245,7 +2250,7 @@ fn microvm_controller_install_intent(revision: ReleaseRevisionBinding) -> Pipeli base_root: microvm_controller_root, revision: revision, ), - microvm_controller_vmm_install_steps(revision: revision), + microvm_controller_vmm_install_steps(revision: revision) + workspace_image_install_steps(revision: revision), ), ), concat( @@ -3564,3 +3569,26 @@ fn workspace_dispatch_grant_steps(revision: ReleaseRevisionBinding) -> List List { + let dir = workspace_installed_image_root(release_dir: microvm_controller_release_dir(revision: revision), identity: workspace_demo_image_identity()) + let kernel = concat(concat(dir, "/"), runner_guest_kernel_artifact.filename as String) + let rootfs = concat(dir, "/workspace.ext4") + match runner_guest_kernel_artifact.integrity { + ArtifactDigestPinned { digest, provenance: _ } => [ + deploy_raw(command: install_d_owned_for_user(user: microvm_controller_owner, path: dir)), + deploy_raw(command: install_file_owned_command(mode: install_file_shared_mode, + owner: microvm_controller_owner, group: microvm_controller_owner, + source: runner_guest_base_artifact_path(root: workspace_image_staging_root, artifact: runner_guest_kernel_artifact), dest: kernel)), + deploy_raw(command: digest_shell_verify_line(digest: digest, file_path: kernel as NonEmptyStr)), + deploy_raw(command: install_file_owned_command(mode: install_file_shared_mode, + owner: microvm_controller_owner, group: microvm_controller_owner, + source: workspace_image_path(root: workspace_image_staging_root), dest: rootfs)), + deploy_raw(command: digest_shell_verify_line(digest: sha256_digest(hex: workspace_demo_image_identity()), file_path: rootfs as NonEmptyStr)), + ] + ArtifactDigestUnpinned { cause: _ } => [deploy_raw(command: "exit 1")] + ArtifactDigestNotPinnable { cause: _ } => [deploy_raw(command: "exit 1")] + } +} diff --git a/dag/gunbc/runner/workspace_guest_image.dag b/dag/gunbc/runner/workspace_guest_image.dag index 73ad7592c79..00441086b6b 100644 --- a/dag/gunbc/runner/workspace_guest_image.dag +++ b/dag/gunbc/runner/workspace_guest_image.dag @@ -196,7 +196,11 @@ fn workspace_installed_image(release_dir: String, identity: NonEmptyStr) -> Work if digest.length() != 64 || !all(digest.chars(), cp => base16_is_lower_digit_code_point(cp: cp)) { WorkspaceImageRefused { reason: "workspace image identity must be its lowercase SHA-256" } } else { - let root = join([release_dir, "/images/workspace/", digest], "") + let root = workspace_installed_image_root(release_dir: release_dir, identity: identity) verify_workspace_image(kernel: concat(concat(root, "/"), runner_guest_kernel_artifact.filename as String), rootfs: concat(root, "/workspace.ext4"), expected_sha256: identity) } } + +fn workspace_installed_image_root(release_dir: String, identity: NonEmptyStr) -> String { + join([release_dir, "/images/workspace/", identity as String], "") +} diff --git a/dag/gunbc/workspace_allocation_dispatch.dag b/dag/gunbc/workspace_allocation_dispatch.dag new file mode 100644 index 00000000000..ee90460625c --- /dev/null +++ b/dag/gunbc/workspace_allocation_dispatch.dag @@ -0,0 +1,132 @@ +module gunbc.workspace_allocation_dispatch + +import std.types { String, NonEmptyStr } +import std.algebra { trim } +import std.process { ExitSuccess, exit_failure } +import extdeps.shell +import extdeps.filesystem.filesystem_io { Filesystem } +import extdeps.languages.json.emit { serialize_json } +import gunbc.cli_run_workspace_root_scaffold { gunbc_workspace_root_env_name } +import gunbc.cli_wire { CliWireResponse, CliWirePrintable } +import gunbc.clock_read { clock_now_probed_at } +import product.placement_supply { HostIdentity } +import gunbc.principal_projection { PrincipalRef } +import gunbc.fabric_storage_binding { fabric_state_storage_binding } +import gunbc.workspace_allocation_convergence { WorkspaceConvergenceObservation, workspace_convergence_observe, WorkspaceConvergenceObserved, WorkspaceConvergenceUnobserved } +import gunbc.workspace_allocation_launch_directive { workspace_launch_directive, workspace_launch_directive_text, workspace_launch_directive_from_text, WorkspaceLaunchDirectiveReady, WorkspaceLaunchDirectiveRefused } +import gunbc.workspace_reconcile_once { workspace_reconcile_prepare } +import gunbc.workspace_allocation_store { WorkspaceAllocation, workspace_allocation_json, workspace_record_preparation_wait, WorkspaceStoreWritten, WorkspaceStoreConflict, WorkspaceStoreWriteRefused } +import gunbc.workspace_request_consumer { workspace_consumer_reconcile_enrolled, WorkspaceConsumerIdle, WorkspaceConsumerRefused, WorkspaceConsumerRetire, WorkspaceConsumerPrepare, WorkspaceConsumerRoute, WorkspaceConsumerSettled, WorkspaceConsumerRetry } +import gunbc.workspace_allocation_policy { workspace_installed_policy } +import gunbc.workspace_allocation_admission { workspace_owner_enrollment, workspace_enrolled_owners, WorkspacePolicyAdmitted, WorkspacePolicyRefused } +import gunbc.workspace_operator_request { workspace_operator_on_candidate_host } + +// Discover the outstanding request from protected state rather than requiring an HTTP process +// to remember or forward its identity. Retirement runs before installed-artifact or offer reads. +fn workspace_allocation_prepare(host: NonEmptyStr, release_dir: NonEmptyStr) -> WorkspaceConvergenceObservation { + match clock_now_probed_at() { + Absent => WorkspaceConvergenceUnobserved { reason: "workspace consumer clock is unavailable" } + Present { value: now } => match workspace_installed_policy() { + WorkspacePolicyRefused { reason: detail } => WorkspaceConvergenceUnobserved { reason: detail } + WorkspacePolicyAdmitted { policy } => match workspace_consumer_reconcile_enrolled(store: fabric_state_storage_binding(), + owners: workspace_enrolled_owners(policy: policy), observed_at: now as String) { + WorkspaceConsumerIdle => WorkspaceConvergenceUnobserved { reason: "no outstanding workspace request" } + WorkspaceConsumerRefused { reason: detail } => WorkspaceConvergenceUnobserved { reason: detail } + WorkspaceConsumerRetry => WorkspaceConvergenceUnobserved { reason: "workspace owner changed concurrently; reconcile again" } + WorkspaceConsumerRetire { allocation: _ } => WorkspaceConvergenceUnobserved { reason: "workspace retirement remains outstanding" } + WorkspaceConsumerSettled { allocation: _ } => WorkspaceConvergenceUnobserved { reason: "unplanned workspace retirement settled; no host apply required" } + WorkspaceConsumerPrepare { allocation: allocation } => match workspace_reconcile_prepare( + store: fabric_state_storage_binding(), allocation: allocation, + local_host: host as HostIdentity, release_dir: release_dir) { + WorkspaceConvergenceUnobserved { reason: detail } => workspace_publish_preparation_wait(allocation: allocation, reason: detail) + WorkspaceConvergenceObserved { basis: planned } => if (planned.host as String) != (host as String) { + WorkspaceConvergenceUnobserved { reason: concat("workspace plan selected another host: ", planned.host as String) } + } else { workspace_allocation_selected(owner: allocation.request.owner, + request_key: allocation.request.idempotency_key, host: host, release_dir: release_dir) } + } + WorkspaceConsumerRoute { allocation: allocation } => workspace_allocation_selected( + owner: allocation.request.owner, request_key: allocation.request.idempotency_key, host: host, release_dir: release_dir) + } + } + } +} + +// A preparation that could not select publishes its located reason for the owner's view and stops. +fn workspace_publish_preparation_wait(allocation: WorkspaceAllocation, reason: String) -> WorkspaceConvergenceObservation { + match workspace_record_preparation_wait(store: fabric_state_storage_binding(), request: allocation.request, reason: reason) { + WorkspaceStoreWritten { allocation: _ } => WorkspaceConvergenceUnobserved { reason: reason } + WorkspaceStoreConflict => WorkspaceConvergenceUnobserved { reason: "workspace changed while publishing preparation wait; reconcile again" } + WorkspaceStoreWriteRefused { reason: detail } => WorkspaceConvergenceUnobserved { reason: detail } + } +} + +// Local authorities are read only after the existing selected-host check. Apply still obtains +// its identity and installed artifact from the hash-admitted plan. Planning binds the locus to +// its admitted run revision, not an independent caller-selected environment path. +fn workspace_allocation_selected(owner: PrincipalRef, request_key: NonEmptyStr, host: NonEmptyStr, release_dir: NonEmptyStr) -> WorkspaceConvergenceObservation { + match workspace_convergence_observe(store: fabric_state_storage_binding(), + owner: owner, idempotency_key: request_key) { + WorkspaceConvergenceUnobserved { reason: detail } => WorkspaceConvergenceUnobserved { reason: detail } + WorkspaceConvergenceObserved { basis: selected } => if selected.host != host { + WorkspaceConvergenceUnobserved { reason: "workspace plan request is executing on a different host" } + } else { WorkspaceConvergenceObserved { basis: selected } } + } +} + +// These entries can only be granted at an exact installed release. Its workspace-root binding is +// part of sudo's exact argv, so stdin cannot select code or an installed image from another tree. +fn workspace_allocation_dispatch_root() -> String { + match shell.Env.Get(name: gunbc_workspace_root_env_name as NonEmptyStr).value { + Present { value } => value + Absent => "" + } +} + +fn workspace_allocation_dispatch_refused(reason: String) -> CliWireResponse { + CliWirePrintable { bytes: "", exit: exit_failure(reason: reason) } +} + +fn workspace_allocation_prepare_cli() -> CliWireResponse { + let input = Filesystem.Read(path: "/dev/stdin") + let fields = split(input.content, separator: "\n") + if !input.success || count(fields) != 2 || !workspace_operator_on_candidate_host() { + workspace_allocation_dispatch_refused(reason: "workspace allocation subject or host is invalid") + } else { match fields.first() { + Absent => workspace_allocation_dispatch_refused(reason: "workspace installed release is absent") + Present { value: installed } => if installed == "" || installed != workspace_allocation_dispatch_root() { + workspace_allocation_dispatch_refused(reason: "workspace allocation release differs from this installed helper") + } else { match fields.last() { + Absent => workspace_allocation_dispatch_refused(reason: "workspace host is absent") + Present { value: host } => match workspace_allocation_prepare(host: host as NonEmptyStr, release_dir: installed as NonEmptyStr) { + WorkspaceConvergenceUnobserved { reason } => workspace_allocation_dispatch_refused(reason: reason) + WorkspaceConvergenceObserved { basis } => CliWirePrintable { + bytes: workspace_launch_directive_text(directive: workspace_launch_directive(basis: basis, release_dir: installed as NonEmptyStr)), exit: ExitSuccess, + } + } + } } + } } +} + +// Apply re-reads only the exact request from the reviewed directive. It never runs preparation or +// chooses a newer request. The controller independently repeats this generation fence at launch. +fn workspace_allocation_readback_cli() -> CliWireResponse { + let input = Filesystem.Read(path: "/dev/stdin") + if !input.success || !workspace_operator_on_candidate_host() { + workspace_allocation_dispatch_refused(reason: "workspace readback input or host is invalid") + } else { match workspace_launch_directive_from_text(text: input.content) { + WorkspaceLaunchDirectiveRefused { reason } => workspace_allocation_dispatch_refused(reason: reason) + WorkspaceLaunchDirectiveReady { directive: planned } => if (planned.release_dir as String) != workspace_allocation_dispatch_root() + || !(match workspace_installed_policy() { + WorkspacePolicyAdmitted { policy } => match workspace_owner_enrollment(policy: policy, owner: planned.allocation.request.owner) { Present { value: _ } => true Absent => false } + WorkspacePolicyRefused { reason: _ } => false + }) { + workspace_allocation_dispatch_refused(reason: "workspace readback release or owner is not admitted") + } else { match workspace_convergence_observe(store: fabric_state_storage_binding(), + owner: planned.allocation.request.owner, idempotency_key: planned.allocation.request.idempotency_key) { + WorkspaceConvergenceUnobserved { reason } => workspace_allocation_dispatch_refused(reason: reason) + WorkspaceConvergenceObserved { basis: current } => CliWirePrintable { + bytes: workspace_launch_directive_text(directive: workspace_launch_directive(basis: current, release_dir: planned.release_dir)), exit: ExitSuccess, + } + } } + } } +} diff --git a/dag/gunbc/workspace_allocation_dispatch_transport.dag b/dag/gunbc/workspace_allocation_dispatch_transport.dag new file mode 100644 index 00000000000..39170ad2123 --- /dev/null +++ b/dag/gunbc/workspace_allocation_dispatch_transport.dag @@ -0,0 +1,51 @@ +module gunbc.workspace_allocation_dispatch_transport + +import std.types { String, NonEmptyStr } +import std.algebra { trim } +import std.durable_compare_and_set { ExpectSlotGeneration } +import extdeps.exec.command { LocalExec } +import extdeps.languages.bash.invocation { bash_program_command } +import extdeps.posix.shell_command_language { posix_single_quote } +import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } +import gunbc.command_runner { run_shell_command_capture } +import gunbc.executor_privileged_operation { executor_privileged_operation_shell_command } +import gunbc.workspace_commissioning_dispatch_operations { workspace_dispatch_operation } +import gunbc.workspace_allocation_store { WorkspaceOwnerSnapshot } +import gunbc.workspace_allocation_convergence { WorkspaceConvergenceObservation, WorkspaceConvergenceUnobserved, workspace_convergence_basis } +import gunbc.workspace_allocation_launch_directive { WorkspaceLaunchDirective, workspace_launch_directive_text, workspace_launch_directive_from_text, WorkspaceLaunchDirectiveReady, WorkspaceLaunchDirectiveRefused } + +fn workspace_allocation_dispatch_command(release_dir: NonEmptyStr, function: NonEmptyStr, payload: String) -> String { + join(["set -o pipefail; printf '%s' ", posix_single_quote(spelling: payload), " | ", + executor_privileged_operation_shell_command(op: workspace_dispatch_operation(release_dir: release_dir, function: function))], "") +} + +fn workspace_allocation_dispatch_result(text: String, host: NonEmptyStr, release_dir: NonEmptyStr) -> WorkspaceConvergenceObservation { + match workspace_launch_directive_from_text(text: trim(s: text)) { + WorkspaceLaunchDirectiveRefused { reason } => WorkspaceConvergenceUnobserved { reason: reason } + WorkspaceLaunchDirectiveReady { directive } => if directive.release_dir != release_dir { + WorkspaceConvergenceUnobserved { reason: "privileged workspace observation returned another installed release" } + } else { workspace_convergence_basis(snapshot: WorkspaceOwnerSnapshot { + owner: directive.allocation.request.owner, expected: ExpectSlotGeneration { generation: directive.owner_generation }, + allocations: [directive.allocation], + }, idempotency_key: directive.allocation.request.idempotency_key, host: host) } + } +} + +fn workspace_allocation_dispatch_call(host: NonEmptyStr, release_dir: NonEmptyStr, function: NonEmptyStr, payload: String) -> WorkspaceConvergenceObservation { + match run_shell_command_capture(command: bash_program_command(script: workspace_allocation_dispatch_command( + release_dir: release_dir, function: function, payload: payload)), transport: LocalExec) { + ProcessOutputPresent { text } => workspace_allocation_dispatch_result(text: text, host: host, release_dir: release_dir) + ProcessOutputAbsent => WorkspaceConvergenceUnobserved { reason: "installed workspace observer returned no receipt" } + ProcessRefused { exit_code: _, stderr } => WorkspaceConvergenceUnobserved { reason: concat("installed workspace observer refused: ", stderr) } + } +} + +fn workspace_allocation_dispatch_prepare(host: NonEmptyStr, release_dir: NonEmptyStr) -> WorkspaceConvergenceObservation { + workspace_allocation_dispatch_call(host: host, release_dir: release_dir, function: "workspace_dispatch_allocation_plan_cli", + payload: join([release_dir as String, host as String], "\n")) +} + +fn workspace_allocation_dispatch_readback(host: NonEmptyStr, planned: WorkspaceLaunchDirective) -> WorkspaceConvergenceObservation { + workspace_allocation_dispatch_call(host: host, release_dir: planned.release_dir, function: "workspace_dispatch_allocation_readback_cli", + payload: workspace_launch_directive_text(directive: planned)) +} diff --git a/dag/gunbc/workspace_artifact_policy.dag b/dag/gunbc/workspace_artifact_policy.dag index 1fa621e2c33..ba286b8bedb 100644 --- a/dag/gunbc/workspace_artifact_policy.dag +++ b/dag/gunbc/workspace_artifact_policy.dag @@ -5,5 +5,5 @@ import std.types { NonEmptyStr } // The pinned workspace guest image the commissioning fleet path installs and admits. It is the // output digest of the pinned-base workspace image builder (gunbc.workspace_guest_image). fn workspace_demo_image_identity() -> NonEmptyStr { - "97859b4f8e206e3398f02dc4070fad14cb5b7a7ec29e305a8895405750044305" + "a64c13977ed6e493e75a6edc3a8b42eb713b1d2de2c3e43759b33550e31af553" } diff --git a/dag/gunbc/workspace_commissioning_dispatch.dag b/dag/gunbc/workspace_commissioning_dispatch.dag index 1caf0234efe..116a5922ff1 100644 --- a/dag/gunbc/workspace_commissioning_dispatch.dag +++ b/dag/gunbc/workspace_commissioning_dispatch.dag @@ -1,5 +1,7 @@ module gunbc.workspace_commissioning_dispatch +import gunbc.workspace_allocation_dispatch { workspace_allocation_prepare_cli, workspace_allocation_readback_cli } + import std.types { String, NonEmptyStr, Bool, CommitSha } import std.decl_ref { decl_ref } import std.algebra { trim } @@ -199,3 +201,7 @@ fn workspace_dispatch_offer_cli() -> CliWireResponse { } } } } + +// Reconcile only already-admitted owner intent; these entries cannot create or release a request. +fn workspace_dispatch_allocation_plan_cli() -> CliWireResponse { workspace_allocation_prepare_cli() } +fn workspace_dispatch_allocation_readback_cli() -> CliWireResponse { workspace_allocation_readback_cli() } diff --git a/dag/gunbc/workspace_commissioning_dispatch_operations.dag b/dag/gunbc/workspace_commissioning_dispatch_operations.dag index d343b81566c..18e3f86d273 100644 --- a/dag/gunbc/workspace_commissioning_dispatch_operations.dag +++ b/dag/gunbc/workspace_commissioning_dispatch_operations.dag @@ -14,7 +14,7 @@ fn workspace_dispatch_operation(release_dir: NonEmptyStr, function: NonEmptyStr) } data workspace_dispatch_grant_path: String = "/etc/sudoers.d/gunbc-workspace-commissioning" -data workspace_dispatch_functions: List = ["workspace_dispatch_plan_cli", "workspace_dispatch_stage_cli", "workspace_dispatch_complete_cli", "workspace_dispatch_offer_cli"] +data workspace_dispatch_functions: List = ["workspace_dispatch_plan_cli", "workspace_dispatch_stage_cli", "workspace_dispatch_complete_cli", "workspace_dispatch_offer_cli", "workspace_dispatch_allocation_plan_cli", "workspace_dispatch_allocation_readback_cli"] fn workspace_dispatch_grant_content(release_dir: NonEmptyStr) -> String { join(flat_map([fleet_generation_writer(), fleet_posix_operator_user.name], grantee => diff --git a/dag/gunbc/workspace_gateway_operator.dag b/dag/gunbc/workspace_gateway_operator.dag new file mode 100644 index 00000000000..634ab51e260 --- /dev/null +++ b/dag/gunbc/workspace_gateway_operator.dag @@ -0,0 +1,76 @@ +module gunbc.workspace_gateway_operator + +import std.types { String, NonEmptyStr, Int, Timestamp } +import std.process { ProcessExit, ExitSuccess, exit_failure } +import std.measure { second } +import std.content_hash { content_hash_of_value, serialize_content_hash } +import extdeps.shell +import extdeps.filesystem.filesystem_io { Filesystem } +import extdeps.tools.sleep { sleep_delay_seconds_second_carrier_projection } +import extdeps.languages.json.emit { serialize_json } +import gunbc.clock_read { clock_now_probed_at } +import gunbc.auth.approval_capability { utc_instant_seconds_after } +import gunbc.fabric_storage_binding { fabric_state_storage_binding } +import gunbc.fleet_known_hosts_anchor { FleetOpenSshTrustPolicy, fleet_ssh_attempt_identity, materialize_fleet_known_hosts_anchor, fleet_openssh_trust_policy_of } +import gunbc.workspace_operator_request { workspace_operator_on_candidate_host, workspace_operator_is_enrolled } +import gunbc.runner_slot_allocation { workspace_execution_slot_identities } +import gunbc.workspace_gateway_observer { workspace_refresh_gateway_wet } +import gunbc.workspace_gateway_store { WorkspaceGatewayEvidenceRead, WorkspaceGatewayEvidenceUnavailable } +import gunbc.workspace_gateway_admission { workspace_gateway_refusal, workspace_gateway_evidence_json } + +// A foreground operator command, bounded to one hour. It maintains the existing 60-second +// forwarding observation while an operator exercises a workspace. It owns no lease or VM +// effects, installs no service or timer, and the root controller never receives the agent. +fn workspace_gateway_refresh_operator_wet(seconds: String) -> ProcessExit { + match parse_int(s: seconds) { + Absent => exit_failure(reason: "GatewayRefreshRefused: canonical duration in seconds required") + Present { value: duration } => if duration < 1 || duration > 3600 || to_string(duration) != seconds { + exit_failure(reason: "GatewayRefreshRefused: duration must be 1..3600 seconds") + } else if !workspace_operator_on_candidate_host() || !workspace_operator_is_enrolled() { + exit_failure(reason: "GatewayRefreshRefused: operator enrollment and workspace host required") + } else { match shell.Env.Get(name: "SSH_AUTH_SOCK").value { + Absent => exit_failure(reason: "GatewayRefreshRefused: select the operator SSH agent") + Present { value: agent } => if agent == "" { exit_failure(reason: "GatewayRefreshRefused: operator agent is empty") } + else { match clock_now_probed_at() { + Absent => exit_failure(reason: "GatewayRefreshRefused: clock unavailable") + Present { value: started } => match fleet_ssh_attempt_identity(raw: concat("workspace-gateway-", serialize_content_hash(hash: content_hash_of_value(value: started)))) { + Absent => exit_failure(reason: "GatewayRefreshRefused: attempt identity invalid") + Present { value: attempt } => match fleet_openssh_trust_policy_of(outcome: materialize_fleet_known_hosts_anchor(attempt: attempt)) { + Absent => exit_failure(reason: "GatewayRefreshRefused: enrolled host keys could not be materialized") + Present { value: trust } => workspace_gateway_refresh_until(started: started, duration: duration, + trust: trust, agent_socket: agent as NonEmptyStr) + } + } + } } + } } + } +} + +fn workspace_gateway_refresh_until(started: Timestamp, duration: Int, trust: FleetOpenSshTrustPolicy, + agent_socket: NonEmptyStr) -> ProcessExit { + match clock_now_probed_at() { + Absent => exit_failure(reason: "GatewayRefreshRefused: clock unavailable") + Present { value: now } => { + let elapsed = utc_instant_seconds_after(later: now, earlier: started) + if elapsed < 0 { exit_failure(reason: "GatewayRefreshRefused: clock moved backwards") } + else if elapsed >= duration { ExitSuccess } + else { + let refusals = flat_map(workspace_execution_slot_identities(), slot => + match workspace_refresh_gateway_wet(store: fabric_state_storage_binding(), slot: slot, trust: trust, agent_socket: agent_socket) { + WorkspaceGatewayEvidenceUnavailable { reason } => [reason as String] + WorkspaceGatewayEvidenceRead { evidence } => match workspace_gateway_refusal(slot: slot, evidence: evidence, now: evidence.observed_at) { + Present { value: reason } => [reason as String] + Absent => { + let receipt = Filesystem.Write(path: "target/workspace-gateway.json", content: concat(serialize_json(v: workspace_gateway_evidence_json(evidence: evidence)), "\n")) + if receipt.success { [] } else { [receipt.error] } + } + } + }) + if count(refusals) != 0 { exit_failure(reason: join(refusals, "; ")) } + else if !sleep_delay_seconds_second_carrier_projection(duration: second(count: 15)) { + exit_failure(reason: "GatewayRefreshRefused: wait failed") + } else { workspace_gateway_refresh_until(started: started, duration: duration, trust: trust, agent_socket: agent_socket) } + } + } + } +} diff --git a/dag/gunbc/workspace_image_delivery.dag b/dag/gunbc/workspace_image_delivery.dag new file mode 100644 index 00000000000..3f8dba50d46 --- /dev/null +++ b/dag/gunbc/workspace_image_delivery.dag @@ -0,0 +1,59 @@ +module gunbc.workspace_image_delivery + +import std.types { String, NonEmptyStr } +import std.process { ProcessExit, ExitSuccess, ExitFailure, exit_failure } +import extdeps.filesystem.filesystem_io { Filesystem } +import extdeps.tools.mkdir { mkdir_parents_command } +import extdeps.tools.curl { curl_download_command } +import extdeps.tools.sha256sum { sha256sum_file_digest_via_shell, Sha256FileDigest, Sha256FileDigestUnavailable } +import gunbc.command_runner { run_shell_commands, LocalExec } +import gunbc.microvm_guest_artifacts { + runner_guest_kernel_artifact, runner_guest_base_rootfs_artifact, runner_guest_base_dir, + runner_guest_base_artifact_path, guest_artifact_verdict, ArtifactMatchesPin, + ArtifactUnpinnedFirstContact, ArtifactBuildReceipt, ArtifactContradictsPin, ArtifactNotMeasured, +} +import gunbc.workspace_guest_image { workspace_image_build, workspace_image_path } + +// Transport is not admission. The administrator installs these exact files, then checks the +// installed kernel and rootfs against the reviewed pins before commissioning can expose supply. +data workspace_image_staging_root: String = "target/workspace-image-artifact" +data workspace_image_artifact_name: String = "workspace-guest-image" +data workspace_image_build_receipt: String = "target/workspace-image-artifact/build-receipt.txt" + +fn workspace_image_artifact_paths() -> String { + join([workspace_image_path(root: workspace_image_staging_root), + runner_guest_base_artifact_path(root: workspace_image_staging_root, artifact: runner_guest_kernel_artifact), + workspace_image_build_receipt], "\n") +} + +// Run the whole entry under fakeroot, so ownership survives extraction and mke2fs -d. +// The digest is a receipt of one build, never a promise that mke2fs output is reproducible. +fn workspace_image_build_wet() -> ProcessExit { + let root = workspace_image_staging_root + match run_shell_commands(commands: [ + mkdir_parents_command(path: runner_guest_base_dir(root: root)), + curl_download_command(url: runner_guest_kernel_artifact.url as String, + destination: runner_guest_base_artifact_path(root: root, artifact: runner_guest_kernel_artifact)), + curl_download_command(url: runner_guest_base_rootfs_artifact.url as String, + destination: runner_guest_base_artifact_path(root: root, artifact: runner_guest_base_rootfs_artifact)), + ], transport: LocalExec) { + ExitFailure { code, reason } => ExitFailure { code: code, reason: reason } + ExitSuccess => match guest_artifact_verdict(artifact: runner_guest_kernel_artifact, + observed: sha256sum_file_digest_via_shell(path: runner_guest_base_artifact_path(root: root, artifact: runner_guest_kernel_artifact) as NonEmptyStr)) { + ArtifactMatchesPin => if !workspace_image_build(root: root) { + exit_failure(reason: "WorkspaceImageBuildRefused: pinned base, SSH tools, ownership or image assembly failed") + } else { match sha256sum_file_digest_via_shell(path: workspace_image_path(root: root) as NonEmptyStr) { + Sha256FileDigestUnavailable { path: _, reason } => exit_failure(reason: reason) + Sha256FileDigest { digest } => { + let receipt = Filesystem.Write(path: workspace_image_build_receipt, + content: join(["workspace-image-sha256=", digest.hex as String, "\n"], "")) + if receipt.success { ExitSuccess } else { exit_failure(reason: receipt.error) } + } + } } + ArtifactUnpinnedFirstContact { digest_hex: _ } => exit_failure(reason: "WorkspaceImageBuildRefused: kernel is unpinned") + ArtifactBuildReceipt { digest_hex: _ } => exit_failure(reason: "WorkspaceImageBuildRefused: kernel is not a pinned input") + ArtifactContradictsPin { expected_hex: _, observed_hex: _ } => exit_failure(reason: "WorkspaceImageBuildRefused: kernel contradicts pin") + ArtifactNotMeasured { reason } => exit_failure(reason: reason) + } + } +} diff --git a/dag/gunbc/workspace_operator_request.dag b/dag/gunbc/workspace_operator_request.dag index 80cfbb14cdb..0fc643cd1bc 100644 --- a/dag/gunbc/workspace_operator_request.dag +++ b/dag/gunbc/workspace_operator_request.dag @@ -1,8 +1,14 @@ module gunbc.workspace_operator_request -import std.types { String, NonEmptyStr, Timestamp } +import std.types { String, NonEmptyStr, Timestamp, Int, Bool } import std.process { ProcessExit, ExitSuccess, exit_failure } import extdeps.shell +import extdeps.filesystem.filesystem_io { Filesystem } +import extdeps.languages.json.emit { serialize_json } +import gunbc.command_runner { run_shell_command_capture, LocalExec } +import extdeps.tools.hostname { hostname_short_read_command } +import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } +import gunbc.runner_slot_allocation { workspace_execution_slot_identities } import gunbc.clock_read { clock_now_probed_at } import gunbc.fabric_storage_binding { fabric_state_storage_binding } import gunbc.fabric_storage_client { FabricStorageBinding } @@ -12,7 +18,7 @@ import gunbc.workspace_allocation_admission { WorkspaceRequestAdmission, workspa import gunbc.workspace_allocation_policy { workspace_operator_principal, workspace_installed_policy } import gunbc.workspace_allocation_admission { workspace_owner_enrollment, WorkspacePolicyAdmitted, WorkspacePolicyRefused } import gunbc.workspace_artifact_policy { workspace_demo_image_identity } -import gunbc.workspace_allocation_store { workspace_record_request_at, WorkspaceStoreWritten, WorkspaceStoreConflict, WorkspaceStoreWriteRefused } +import gunbc.workspace_allocation_store { workspace_record_request_at, WorkspaceStoreWritten, WorkspaceStoreConflict, WorkspaceStoreWriteRefused, WorkspaceAllocation, workspace_allocation_json, workspace_read_owner, WorkspaceStoreReadOk, WorkspaceStoreUnreadable, workspace_request_release_at } // OPERATOR-ONLY FIRST REVISION. The owner is the single enrolled operator principal, not a // caller-supplied identity: this producer exists so a first request can be recorded while the @@ -42,7 +48,7 @@ fn workspace_operator_request_record(store: FabricStorageBinding, observed_at: T WorkspaceRequestPolicyRefused { reason: detail } => exit_failure(reason: concat("WorkspaceOperatorRequestRefused: ", detail as String)) WorkspaceRequestAdmissionRefused { decision: _ } => exit_failure(reason: "WorkspaceOperatorRequestRefused: operator admission refused") WorkspaceRequestAdmitted { request: admitted } => match workspace_record_request_at(store: store, request: admitted, observed_at: observed_at) { - WorkspaceStoreWritten { allocation: _ } => ExitSuccess + WorkspaceStoreWritten { allocation } => workspace_operator_receipt(allocation: allocation) WorkspaceStoreConflict => exit_failure(reason: "WorkspaceOperatorRequestConflict: owner ledger changed concurrently; resubmit the same key") WorkspaceStoreWriteRefused { reason: detail } => exit_failure(reason: concat("WorkspaceOperatorRequestRefused: ", detail)) } @@ -55,12 +61,77 @@ fn workspace_operator_request_env(name: NonEmptyStr) -> String { // Typed entry: key, profile and lease end are the only operator inputs. fn workspace_operator_request_wet() -> ProcessExit { - match clock_now_probed_at() { + if !workspace_operator_on_candidate_host() { exit_failure(reason: "WorkspaceOperatorRequestRefused: executor is not a workspace host") } + else { match clock_now_probed_at() { Absent => exit_failure(reason: "WorkspaceOperatorRequestRefused: request clock is unavailable") Present { value: now } => workspace_operator_request_record(store: fabric_state_storage_binding(), observed_at: now as Timestamp, key: workspace_operator_request_env(name: "GUNBC_WORKSPACE_REQUEST_KEY"), profile_wire: workspace_operator_request_env(name: "GUNBC_WORKSPACE_REQUEST_PROFILE"), expires_at: workspace_operator_request_env(name: "GUNBC_WORKSPACE_REQUEST_EXPIRES_AT") as Timestamp) + } } +} + +// These entries only publish intent to the protected owner ledger. The existing reconciler owns +// reservation, VM effects and cleanup; a successful intent write never claims a usable VM. +data workspace_operator_receipt_path: String = "target/workspace-operation.json" + +fn workspace_operator_on_candidate_host() -> Bool { + match run_shell_command_capture(command: hostname_short_read_command(), transport: LocalExec) { + ProcessOutputPresent { text } => any(workspace_execution_slot_identities(), slot => (slot.host as String) == trim(s: text)) + ProcessOutputAbsent => false + ProcessRefused { exit_code: _, stderr: _ } => false + } +} + +fn workspace_operator_receipt(allocation: WorkspaceAllocation) -> ProcessExit { + let written = Filesystem.Write(path: workspace_operator_receipt_path, + content: concat(serialize_json(v: workspace_allocation_json(allocation: allocation)), "\n")) + if written.success { ExitSuccess } else { exit_failure(reason: written.error) } +} + +fn workspace_operator_release_wet() -> ProcessExit { + let key = workspace_operator_request_env(name: "GUNBC_WORKSPACE_REQUEST_KEY") + let generation = workspace_operator_request_env(name: "GUNBC_WORKSPACE_RELEASE_GENERATION") + if !workspace_operator_on_candidate_host() { exit_failure(reason: "WorkspaceOperatorReleaseRefused: executor is not a workspace host") } + else if !workspace_operator_is_enrolled() { exit_failure(reason: "WorkspaceOperatorReleaseRefused: operator is not enrolled") } + else { match workspace_allocation_id_of(raw: key) { + Absent => exit_failure(reason: "WorkspaceOperatorReleaseRefused: invalid request key") + Present { value: id } => match parse_int(s: generation) { + Absent => exit_failure(reason: "WorkspaceOperatorReleaseRefused: canonical reservation generation required") + Present { value: expected } => if expected < 0 || to_string(expected) != generation { + exit_failure(reason: "WorkspaceOperatorReleaseRefused: canonical nonnegative generation required") + } else { match clock_now_probed_at() { + Absent => exit_failure(reason: "WorkspaceOperatorReleaseRefused: clock unavailable") + Present { value: now } => match workspace_request_release_at(store: fabric_state_storage_binding(), + owner: workspace_operator_principal(), idempotency_key: id, expected_generation: expected as Int, + observed_at: now as Timestamp) { + WorkspaceStoreWritten { allocation } => workspace_operator_receipt(allocation: allocation) + WorkspaceStoreConflict => exit_failure(reason: "WorkspaceOperatorReleaseConflict: retry the same request and generation") + WorkspaceStoreWriteRefused { reason } => exit_failure(reason: reason) + } + } } + } + } } +} + +fn workspace_operator_observe_wet() -> ProcessExit { + let key = workspace_operator_request_env(name: "GUNBC_WORKSPACE_REQUEST_KEY") + match workspace_read_owner(store: fabric_state_storage_binding(), owner: workspace_operator_principal()) { + WorkspaceStoreUnreadable { reason } => exit_failure(reason: reason) + WorkspaceStoreReadOk { snapshot } => match filter(snapshot.allocations, a => (a.request.idempotency_key as String) == key).first() { + Absent => exit_failure(reason: "WorkspaceOperatorObserveRefused: request absent for operator") + Present { value: allocation } => workspace_operator_receipt(allocation: allocation) + } + } +} + +fn workspace_operator_is_enrolled() -> Bool { + match workspace_installed_policy() { + WorkspacePolicyRefused { reason: _ } => false + WorkspacePolicyAdmitted { policy } => match workspace_owner_enrollment(policy: policy, owner: workspace_operator_principal()) { + Absent => false + Present { value: _ } => true + } } } diff --git a/dag/test/claim/workspace_allocation_dispatch_witness_test.dag b/dag/test/claim/workspace_allocation_dispatch_witness_test.dag new file mode 100644 index 00000000000..f0c2a6c9610 --- /dev/null +++ b/dag/test/claim/workspace_allocation_dispatch_witness_test.dag @@ -0,0 +1,46 @@ +module test.claim.workspace_allocation_dispatch_witness_test + +import std.types { Bool, String, NonEmptyStr } +import gunbc.workspace_allocation_dispatch_transport { workspace_allocation_dispatch_result, workspace_allocation_dispatch_command } +import gunbc.workspace_allocation_dispatch { workspace_allocation_prepare_cli, workspace_allocation_readback_cli } +import gunbc.workspace_allocation_convergence { WorkspaceConvergenceObserved, WorkspaceConvergenceUnobserved } +import gunbc.workspace_allocation_launch_directive { workspace_launch_directive_text, WorkspaceLaunchDirectiveReady } +import gunbc.test.claim.workspace_launch_directive_witness_test { launch_fixture_allocation_in_slot, launch_fixture_directive } +import gunbc.workspace_commissioning_dispatch_operations { workspace_dispatch_grant_content } + +fn dispatch_result_admits(host: NonEmptyStr, installed: NonEmptyStr) -> Bool { + match launch_fixture_allocation_in_slot(index: 13) { + Absent => false + Present { value: allocation } => match launch_fixture_directive(allocation: allocation) { + WorkspaceLaunchDirectiveReady { directive } => match workspace_allocation_dispatch_result( + text: workspace_launch_directive_text(directive: directive), host: host, release_dir: installed) { + WorkspaceConvergenceObserved { basis } => basis.owner_generation == 4 && basis.allocation == allocation + WorkspaceConvergenceUnobserved { reason: _ } => false + } + other => false + } + } +} + +test fn installed_observation_binds_host_release_and_owner_generation() -> Bool { + dispatch_result_admits(host: "srv1", installed: "/opt/gunbc/controller/fixture-revision") + && !dispatch_result_admits(host: "srv2", installed: "/opt/gunbc/controller/fixture-revision") + && !dispatch_result_admits(host: "srv1", installed: "/opt/gunbc/controller/other-revision") +} + +test fn installed_observation_refuses_empty_and_malformed_responses() -> Bool { + all(["", "not-json", "\{}"], text => match workspace_allocation_dispatch_result( + text: text, host: "srv1", release_dir: "/opt/gunbc/controller/fixture-revision") { + WorkspaceConvergenceUnobserved { reason: _ } => true + WorkspaceConvergenceObserved { basis: _ } => false + }) +} + +test fn runner_grants_reconcile_existing_intent_without_operator_write_authority() -> Bool { + let grants = workspace_dispatch_grant_content(release_dir: "/opt/gunbc/controller/fixture-revision") + string_contains(s: grants, pattern: "workspace_dispatch_allocation_plan_cli") + && string_contains(s: grants, pattern: "workspace_dispatch_allocation_readback_cli") + && !string_contains(s: grants, pattern: "workspace_operator_request") + && !string_contains(s: grants, pattern: "workspace_operator_release") + && !string_contains(s: grants, pattern: "state-writer-mac-key") +} diff --git a/dag/test/claim/workspace_gateway_operator_witness_test.dag b/dag/test/claim/workspace_gateway_operator_witness_test.dag new file mode 100644 index 00000000000..a91e179fa21 --- /dev/null +++ b/dag/test/claim/workspace_gateway_operator_witness_test.dag @@ -0,0 +1,12 @@ +module test.claim.workspace_gateway_operator_witness_test + +import std.types { Bool } +import std.process { ExitSuccess, ExitFailure } +import gunbc.workspace_gateway_operator { workspace_gateway_refresh_operator_wet } + +test fn gateway_refresh_refuses_unbounded_or_noncanonical_durations_before_observation() -> Bool { + all(["", "0", "-1", "3601", "01", "forever"], duration => match workspace_gateway_refresh_operator_wet(seconds: duration) { + ExitFailure { code: _, reason } => starts_with(s: reason, prefix: "GatewayRefreshRefused:") + ExitSuccess => false + }) +} diff --git a/dag/test/claim/workspace_launch_directive_witness_test.dag b/dag/test/claim/workspace_launch_directive_witness_test.dag index dd0f9d60cdf..e248ee51a1b 100644 --- a/dag/test/claim/workspace_launch_directive_witness_test.dag +++ b/dag/test/claim/workspace_launch_directive_witness_test.dag @@ -22,13 +22,15 @@ import gunbc.fabric_control_plane { fabric_money_reservation_ref } import gunbc.workspace_allocation_release { workspace_release_derive, workspace_release_wire, workspace_release_decode, WorkspaceReleasePrepared, WorkspaceReleaseRefused } import gunbc.workspace_money_codec { workspace_money_json } -fn launch_fixture_allocation() -> WorkspaceAllocation? { +fn launch_fixture_allocation() -> WorkspaceAllocation? { launch_fixture_allocation_in_slot(index: 7) } + +fn launch_fixture_allocation_in_slot(index: Int) -> WorkspaceAllocation? { let request = allocation_test_request(key: "launch-fixture" as NonEmptyStr) let work = workspace_work(request: request).id match workspace_memory_grant(request: request, work: work, attempt: FabricIdentity { principal: work.principal, key: "fixture-attempt" as ExecutionAttemptKey }) { MemoryGrantIssued { grant: committed } => { let reserved = WorkspaceReservation { - request: request, slot: RunnerSlotIdentity { host: "srv1" as HostIdentity, slot_index: 7 }, generation: 1, + request: request, slot: RunnerSlotIdentity { host: "srv1" as HostIdentity, slot_index: index }, generation: 1, grant: committed, work: work, demand: FabricIdentity { principal: work.principal, key: "fixture-demand" as DemandKey }, offer: FabricIdentity { principal: "fixture-provider", key: "fixture-offer" as OfferKey }, diff --git a/dag/test/claim/workspace_workflow_dispatch_witness_test.dag b/dag/test/claim/workspace_workflow_dispatch_witness_test.dag new file mode 100644 index 00000000000..a4829300beb --- /dev/null +++ b/dag/test/claim/workspace_workflow_dispatch_witness_test.dag @@ -0,0 +1,38 @@ +module test.claim.workspace_workflow_dispatch_witness_test + +import std.types { Bool } +import gunbc.fleet_converge_workflow { + WorkspaceAllocationPlan, WorkspaceImageBuild, + fleet_converge_mode_scope, fleet_converge_mode_job_id, fleet_converge_shared_job_id, + fleet_converge_mode_release_product, CompilerPairOnly, + fleet_converge_mode_fleet_ssh_key_demand, FleetSshKeyNotConsumed, +} +import gunbc.fleet_converge_scope { WorkspaceAllocationOnly } +import gunbc.ci_spec { gunbc_ci_workspace_image_build_invoke } +import gunbc.workspace_image_delivery { workspace_image_artifact_paths } + +test fn workspace_allocation_uses_its_existing_scope() -> Bool { + (match fleet_converge_mode_scope(mode: WorkspaceAllocationPlan) { + Present { value: WorkspaceAllocationOnly } => true + Present { value: _ } => false + Absent => false + }) && all([WorkspaceImageBuild], mode => + match fleet_converge_mode_scope(mode: mode) { Absent => true Present { value: _ } => false }) +} + +test fn workspace_operations_use_the_serialized_host_job_without_fleet_credentials() -> Bool { + all([WorkspaceAllocationPlan, WorkspaceImageBuild], mode => + fleet_converge_mode_job_id(mode: mode) == fleet_converge_shared_job_id + && fleet_converge_mode_release_product(mode: mode) == CompilerPairOnly + && fleet_converge_mode_fleet_ssh_key_demand(mode: mode) == FleetSshKeyNotConsumed) +} + +test fn workspace_image_build_preserves_ownership_and_archives_only_the_product() -> Bool { + let build = gunbc_ci_workspace_image_build_invoke() + let paths = workspace_image_artifact_paths() + string_contains(s: build, pattern: "fakeroot -- ") + && string_contains(s: build, pattern: "workspace_image_build_wet") + && string_contains(s: paths, pattern: "/workspace.ext4") + && !string_contains(s: paths, pattern: "workspace-tree") + && !string_contains(s: paths, pattern: "squashfs") +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md new file mode 100644 index 00000000000..5eb3735acf6 --- /dev/null +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -0,0 +1,81 @@ +# srv1 workspace VM bringup + +Scope: the operator-owned workspace on `srv1-13`, using the existing commissioning, +allocation, controller and cleanup protocols. No customer create API, additional +hosts, CI-runner migration, or automatic expiry scheduling is part of this change. + +## Observed gaps + +The October 9, 2026 inspection found no successful workspace VM lifecycle receipt. +The September 24 controller invocation `c0fceefc7d9643b0aaff0617907b1335` +refused because it could not read the allocation store for `srv1-13`. +[Run 36054790165](https://github.com/gunb-ai/gunbc/actions/runs/36054790165) +collected that refusal; its green workflow conclusion is not a boot receipt. + +| Boundary | Observed state | Required change | +| --- | --- | --- | +| Guest image | Builder exists; no image delivery in controller install | Build under fakeroot, archive kernel/rootfs/digest receipt, pin the measured rootfs, verify installed bytes and custody | +| Controller | Installed revision `cfb9ff80652fe0a74093f08cade64f59d9695266` | Install this PR through the existing controller installer | +| Slot budget | `fabric-cell-srv1-13.slice` has no installed unit file | Converge the modeled execution-cell boundary; a synthesized systemd slice is not proof | +| Commissioning | No commissioning or readiness record | Run reviewed commissioning plan/apply and verify its protected transaction | +| Protected state service | Running revision `5b96cab749088aa5cec71ce9ad52e1857724d252` predates the authenticated state route | Converge fabric storage through its existing deployment path | +| State-key custody | Key exists; operator can read it; `ghrunner` cannot | Preserve custody; exact installed helpers reconcile admitted intent and read the selected allocation | +| Gateway observation | Observer has no production caller; admission expires after 60 seconds | Bounded foreground operator refresh using the enrolled agent and host keys | +| Allocation dispatch | No workflow mode; release path came from an unrelated environment variable | Add `workspace_allocation_plan`, bind its installed release to the admitted run revision, reuse reviewed `apply` | +| Operator lifecycle | Request producer exists; release/observation need CLI entries | Add generation-checked release and owner-scoped JSON receipts under operator authority | + +The historical refusal files are evidence. Do not delete them to manufacture an +empty commissioning prestate. Do not restore the old CI reservation service: +`srv1-13` is a workspace slot. + +## Operation + +1. Dispatch `fleet-converge`, `host=srv1`, `mode=workspace_image_build` at the + candidate revision. The `workspace-guest-image` artifact contains only the + kernel, rootfs and measured digest receipt. The filesystem build is not + reproducible: pin that build's measured digest in `workspace_artifact_policy`. +2. Dispatch `microvm_controller_install` at the reviewed revision with + `workspace_image_run_id` naming that build. The run ID locates bytes; it does + not authorize them. The installer checks the reviewed kernel and rootfs pins + after copying into the root-owned release directory, then independently reads + back the image and installed executor. +3. Converge the execution-cell boundary and authenticated state service, then run + `workspace_commissioning_plan` and its reviewed `apply` on srv1. The existing + operator-local fleet plan/apply path is available for pre-merge validation; + do not widen the main-only GCP login to run branch workflows. +4. Keep `workspace_gateway_refresh_operator_wet --arg seconds=3600` running in + the operator context with the enrolled key selected in `SSH_AUTH_SOCK`. It + refreshes forwarding evidence every 15 seconds and stops on refusal or at its + deadline. It does not install a scheduler. As the enrolled operator on srv1, use `workspace_operator_request_wet` with + `GUNBC_WORKSPACE_REQUEST_KEY`, `GUNBC_WORKSPACE_REQUEST_PROFILE` and + `GUNBC_WORKSPACE_REQUEST_EXPIRES_AT`. Use a canonical UTC expiry within the + policy's one-hour lease. The owner and public key come from installed policy. +5. Dispatch `workspace_allocation_plan` and review/apply its artifact. Observe + through `workspace_operator_observe_wet` with the same key. Each operator + command writes `target/workspace-operation.json`; a recorded request alone + is not evidence that the guest is usable. +6. Authenticate SSH using the allocation's connection data and verified host key. + Record the guest identity and a command executed in the guest. +7. Run `workspace_operator_release_wet` with the key and + `GUNBC_WORKSPACE_RELEASE_GENERATION` from the observation. Reconcile with + another allocation plan/apply, and verify released state and absent guest. +8. Request a second key, prove the same slot is reused with a new incarnation, + and let its short lease expire. Explicitly dispatch allocation plan/apply to + clean it up. No timer or scheduled expiry sweep is installed by this PR. + +Workflow jobs receive no state MAC key and no grant to create or release an +operator request. Their installed helper can act only on already-admitted ledger +intent. Apply re-reads the exact request and generation carried in the reviewed +plan; it does not select the latest request again. + +## Acceptance evidence + +Keep the PR draft until all of the following have live receipts: image/controller +readback, commissioned slot, ready allocation, authenticated SSH, explicit release, +same-slot reuse with a new incarnation, expiry cleanup, and final released/ready +state. Record run IDs, revisions, allocation IDs and generations here as they are +observed. Never record private keys, tokens or signed state-request envelopes. + +Local validation so far: the image builder produced an ext4 image whose `/root` +and `/usr/sbin/sshd` are owned by UID/GID 0, and the fresh compiler parsed the full +source tree. Live acceptance is pending. From 521085ab254290029f8dfbaa312a218f84a6be7b Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 03:14:08 +0000 Subject: [PATCH 02/19] Keep fleet dispatch within GitHub limits and enroll the workspace gateway --- .github/workflows/fleet-converge.yml | 32 ++++--------------- dag/extdeps/github/actions.dag | 4 +++ dag/gunbc/fleet/fleet_converge_workflow.dag | 24 ++++++++------ dag/gunbc/fleet/fleet_host_key_enrollment.dag | 8 +++++ .../receipt_consumer_without_producer.dag | 1 + ...orkflow_dispatch_input_limit_unchecked.dag | 14 ++++++++ ...orkspace_gateway_operator_witness_test.dag | 13 +++++++- ...rkspace_workflow_dispatch_witness_test.dag | 6 +++- docs/plans/srv1-workspace-vm-bringup.md | 7 ++++ 9 files changed, 72 insertions(+), 37 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/workflow_dispatch_input_limit_unchecked.dag diff --git a/.github/workflows/fleet-converge.yml b/.github/workflows/fleet-converge.yml index 9384d957d26..df414398708 100644 --- a/.github/workflows/fleet-converge.yml +++ b/.github/workflows/fleet-converge.yml @@ -100,28 +100,10 @@ on: required: false options: [printer-01, printer-02] type: choice - printer_operation: - description: Observe credential identity or start one pinned project - required: false - options: [observe, print] - type: choice - printer_project_path: - description: Absolute path of staged sliced project on srv1 - required: false - type: string - printer_project_sha256: - description: SHA-256 of the reviewed sliced project + printer_request: + description: "Printer operation JSON: operation (observe/print), project_path, project_sha256, credential_version, bed_clear (string true/false). The existing printer admission validates every field." required: false type: string - printer_credential_version: - description: Numeric Secret Manager version returned by observation - required: false - type: string - printer_bed_clear: - description: Operator reports this printer bed is clear for this dispatch - required: false - options: ["false", "true"] - type: choice workspace_source_object: description: "workspace_checkpoint_measure only: the lower-hex SHA-256 the workspace_source_pack receipt printed as address= (the archive's own digest; the fetched bytes must hash to it)" required: false @@ -3810,11 +3792,11 @@ jobs: GUNBC_APPROVAL_CAPABILITY_MAC_KEY_FILE: /etc/gunbc-roadmap/approval-mac-key GUNBC_APPROVAL_STORE_RECEIPT_MAC_KEY_FILE: /etc/gunbc-roadmap/approval-store-receipt-mac-key PRINTER_TARGET: ${{ inputs.printer }} - PRINTER_OPERATION: ${{ inputs.printer_operation }} - PRINTER_PROJECT_PATH: ${{ inputs.printer_project_path }} - PRINTER_PROJECT_SHA256: ${{ inputs.printer_project_sha256 }} - PRINTER_CREDENTIAL_VERSION: ${{ inputs.printer_credential_version }} - PRINTER_BED_CLEAR: ${{ inputs.printer_bed_clear }} + PRINTER_OPERATION: ${{ fromJSON(inputs.printer_request || '{}').operation }} + PRINTER_PROJECT_PATH: ${{ fromJSON(inputs.printer_request || '{}').project_path }} + PRINTER_PROJECT_SHA256: ${{ fromJSON(inputs.printer_request || '{}').project_sha256 }} + PRINTER_CREDENTIAL_VERSION: ${{ fromJSON(inputs.printer_request || '{}').credential_version }} + PRINTER_BED_CLEAR: ${{ fromJSON(inputs.printer_request || '{}').bed_clear }} timeout-minutes: 15 - name: Upload printer operation receipt id: printer_receipt diff --git a/dag/extdeps/github/actions.dag b/dag/extdeps/github/actions.dag index 1e37730c509..bb250c9135f 100644 --- a/dag/extdeps/github/actions.dag +++ b/dag/extdeps/github/actions.dag @@ -411,3 +411,7 @@ data max_matrix_combinations: Int = 256 // repository's own live ruleset row (GET /repos/gunb-ai/gunbc/rulesets, integration_id 15368, // 2026-08-27) and consistent with the documented app id for GitHub Actions. data github_actions_app_integration_id: Int = 15368 + +// https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#onworkflow_dispatchinputs +// GitHub increased the top-level workflow_dispatch input limit from 10 to 25 on 2025-12-04. +data workflow_dispatch_max_inputs: Int = 25 diff --git a/dag/gunbc/fleet/fleet_converge_workflow.dag b/dag/gunbc/fleet/fleet_converge_workflow.dag index 9c3c09e4d59..61c219b3bb0 100644 --- a/dag/gunbc/fleet/fleet_converge_workflow.dag +++ b/dag/gunbc/fleet/fleet_converge_workflow.dag @@ -1,5 +1,7 @@ module gunbc.fleet_converge_workflow +import extdeps.github.actions { workflow_dispatch_max_inputs } + import gunbc.ci_spec { gunbc_ci_workspace_allocation_plan_invoke, gunbc_ci_workspace_image_build_invoke } import gunbc.workspace_image_delivery { workspace_image_artifact_name, workspace_image_staging_root, workspace_image_artifact_paths } @@ -5135,11 +5137,7 @@ data fleet_converge_dispatch_inputs: List = [ type: InputString, }, DispatchInput { name: "printer", description: Present { value: "Printer identity; required for printer mode" }, required: false, default: none, type: InputChoice { options: map(printer_access_code_loci,p=>p.printer as String) } }, - DispatchInput { name: "printer_operation", description: Present { value: "Observe credential identity or start one pinned project" }, required: false, default: none, type: InputChoice { options: ["observe", "print"] } }, - DispatchInput { name: "printer_project_path", description: Present { value: "Absolute path of staged sliced project on srv1" }, required: false, default: none, type: InputString }, - DispatchInput { name: "printer_project_sha256", description: Present { value: "SHA-256 of the reviewed sliced project" }, required: false, default: none, type: InputString }, - DispatchInput { name: "printer_credential_version", description: Present { value: "Numeric Secret Manager version returned by observation" }, required: false, default: none, type: InputString }, - DispatchInput { name: "printer_bed_clear", description: Present { value: "Operator reports this printer bed is clear for this dispatch" }, required: false, default: none, type: InputChoice { options: ["false", "true"] } }, + DispatchInput { name: "printer_request", description: Present { value: "Printer operation JSON: operation (observe/print), project_path, project_sha256, credential_version, bed_clear (string true/false). The existing printer admission validates every field." }, required: false, default: none, type: InputString }, DispatchInput { name: "workspace_source_object", description: Present { value: "workspace_checkpoint_measure only: the lower-hex SHA-256 the workspace_source_pack receipt printed as address= (the archive's own digest; the fetched bytes must hash to it)" }, @@ -5194,6 +5192,12 @@ type FleetConvergeYamlGenerationOutcome data fleet_converge_capability_closure_refusal: String = "REFUSED gunbc.fleet_converge_workflow: a step consumes a capability that no preceding step provides (gunbc.workflow_capability_closure). No workflow yaml was emitted. Fix the step roles in ci_prelude_bound_steps_with_checkout / fleet_converge_build_job_annotations, or the order of the steps themselves." fn expected_fleet_converge_yml() -> FleetConvergeYamlGenerationOutcome { + if count(fleet_converge_dispatch_inputs) > workflow_dispatch_max_inputs { + FleetConvergeYamlGenerationRefused { reason: "fleet-converge exceeds GitHub's workflow_dispatch input limit" } + } else { fleet_converge_yml_after_input_admission() } +} + +fn fleet_converge_yml_after_input_admission() -> FleetConvergeYamlGenerationOutcome { match admit_workflow_action_uses(workflow: fleet_converge_workflow) { WorkflowActionUsesRefused { refusals: rs } => FleetConvergeYamlGenerationRefused { reason: workflow_action_uses_refusal_reason(workflow_module: "gunbc.fleet_converge_workflow", refusals: rs) } @@ -5345,11 +5349,11 @@ fn fleet_converge_printer_job() -> Job { kv(key:approval_capability_mac_key_path_env as String,value:yaml_string(s:approval_mac_key_path as String)), kv(key:approval_store_receipt_mac_key_path_env as String,value:yaml_string(s:approval_store_receipt_mac_key_path as String)), kv(key: "PRINTER_TARGET", value: yaml_string(s: "${{ inputs.printer }}")), - kv(key: "PRINTER_OPERATION", value: yaml_string(s: "${{ inputs.printer_operation }}")), - kv(key: "PRINTER_PROJECT_PATH", value: yaml_string(s: "${{ inputs.printer_project_path }}")), - kv(key: "PRINTER_PROJECT_SHA256", value: yaml_string(s: "${{ inputs.printer_project_sha256 }}")), - kv(key: "PRINTER_CREDENTIAL_VERSION", value: yaml_string(s: "${{ inputs.printer_credential_version }}")), - kv(key: "PRINTER_BED_CLEAR", value: yaml_string(s: "${{ inputs.printer_bed_clear }}")), + kv(key: "PRINTER_OPERATION", value: yaml_string(s: "${{ fromJSON(inputs.printer_request || '\{}').operation }}")), + kv(key: "PRINTER_PROJECT_PATH", value: yaml_string(s: "${{ fromJSON(inputs.printer_request || '\{}').project_path }}")), + kv(key: "PRINTER_PROJECT_SHA256", value: yaml_string(s: "${{ fromJSON(inputs.printer_request || '\{}').project_sha256 }}")), + kv(key: "PRINTER_CREDENTIAL_VERSION", value: yaml_string(s: "${{ fromJSON(inputs.printer_request || '\{}').credential_version }}")), + kv(key: "PRINTER_BED_CLEAR", value: yaml_string(s: "${{ fromJSON(inputs.printer_request || '\{}').bed_clear }}")), ] }, working_directory: none, if_condition: none, continue_on_error: none, timeout_minutes: Present { value: gunbc_ci_gunbc_run_resolve_step_timeout_minutes }, diff --git a/dag/gunbc/fleet/fleet_host_key_enrollment.dag b/dag/gunbc/fleet/fleet_host_key_enrollment.dag index aeac66068a3..2e3db9c35e2 100644 --- a/dag/gunbc/fleet/fleet_host_key_enrollment.dag +++ b/dag/gunbc/fleet/fleet_host_key_enrollment.dag @@ -108,6 +108,14 @@ data fleet_host_key_enrollments: List = [ sha256_fingerprint: "SHA256:hIxG0YT2jIxXbxs4XGqwehgoA63kxD4xKP2j6YjG86w", provenance: ScannedFirstContact { scanned_from: "srv1 via ssh-keyscan -t ed25519 from the LAN probe position; no cited reservation, procurement, or external key authority exists" as NonEmptyStr }, }, + KnownHostKeyRow { + host: "srv1" as HostIdentity, + endpoint: "srv1.tailecbe08.ts.net", + key_type: "ssh-ed25519", + public_key_base64: "AAAAC3NzaC1lZDI1NTE5AAAAIErU2itxhvGPRx1UxiuCYkikYIV9DtuXiELy64bQpAXL", + sha256_fingerprint: "SHA256:hIxG0YT2jIxXbxs4XGqwehgoA63kxD4xKP2j6YjG86w", + provenance: ScannedFirstContact { scanned_from: "2026-10-09 srv2 ssh-keyscan of srv1.tailecbe08.ts.net matched the existing srv1 enrollment byte-for-byte; the authenticated srv1 SSH session independently reported this DNSName through tailscale status" }, + }, KnownHostKeyRow { host: "srv2" as HostIdentity, endpoint: "srv2", diff --git a/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag b/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag index 12437681cc7..964d3116d25 100644 --- a/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag +++ b/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag @@ -6,6 +6,7 @@ import gunbc.recurring_failure_mode { RecurringFailureMode } data receipt_consumer_without_producer: RecurringFailureMode = RecurringFailureMode { identity: "receipt_consumer_without_producer" as NonEmptyStr, receipts: [ + "2026-10-09 srv1 workspace bringup, PR #13607: allocation offer admission and the controller require a gateway forwarding observation no older than 60 seconds, but workspace_refresh_gateway_wet had no production caller. The observer existed and its decoder was tested, while every real offer necessarily refused an absent or stale receipt. A bounded foreground operator command now calls the existing observer and protected publication repeatedly using the enrolled agent and host keys; no scheduler or private-key copy is introduced. Acceptance still requires the live forwarding receipt and authenticated guest SSH, not just the command returning success.", "PR #12421 review 5331195485: gunbc_ci_namecheap_observe_invoke supplied receipt_rel as though it captured execution output, but gunbc_invoke_receipt_steps only read an existing file. A successful observer therefore failed the shell step and suppressed its artifact upload. Invalid state: an emitted receipt read with no producer. Harm: successful observation becomes workflow failure. Discriminating facts: execute emitted shell in a clean root with only the declared output; a planted unmatched read must fail. Mitigation: writer, console display and uploader derive the path from one declaration. Ceiling: workflow construction rejects a receipt consumer without a producer edge; next trigger: that consumed workflow admission relation.", ], evidence: [], diff --git a/dag/gunbc/recurring_failure_mode/workflow_dispatch_input_limit_unchecked.dag b/dag/gunbc/recurring_failure_mode/workflow_dispatch_input_limit_unchecked.dag new file mode 100644 index 00000000000..6811ca37cb8 --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/workflow_dispatch_input_limit_unchecked.dag @@ -0,0 +1,14 @@ +module gunbc.recurring_failure_mode.workflow_dispatch_input_limit_unchecked + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data workflow_dispatch_input_limit_unchecked: RecurringFailureMode = RecurringFailureMode { + identity: "workflow_dispatch_input_limit_unchecked" as NonEmptyStr, + receipts: [ + "INVALID STATE: a generated workflow is valid YAML and passes local generation but declares more workflow_dispatch inputs than GitHub accepts. Every manual mode is then unavailable, including modes that use only a few fields. A YAML parser cannot prove the hosted API's cardinality contract.", + "OBSERVED 2026-10-09 during srv1 workspace bringup, PR #13607: base 2a519d27f6f77db2a20696a2ffc4f540dbd88f40 declares 28 inputs. Adding the image artifact locator made 29; dispatch at 048d345cd302d412565d6b214f1e912a788ffecb returned HTTP 422 with 'you may only define up to 25 inputs'. Push run 37877157911 also failed workflow validation, before any host job ran. GitHub documents 25 at https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#onworkflow_dispatchinputs .", + "REPAIR: extdeps.github.actions owns workflow_dispatch_max_inputs, and the fleet YAML producer refuses above it. Consolidating five printer operation detail inputs into one JSON input retains their domain checks and brings this workflow to 25 inputs including the workspace artifact locator. RUNG FOUND AT: detectable only by the remote dispatcher. CURRENT RUNG: mechanically preventable for this producer. CEILING: structurally impossible when every workflow-dispatch carrier admits its bounded input population at construction. NEXT TRIGGER: generic workflow construction owns this admission, so another emitter cannot omit the same API limit.", + ], + evidence: [], +} diff --git a/dag/test/claim/workspace_gateway_operator_witness_test.dag b/dag/test/claim/workspace_gateway_operator_witness_test.dag index a91e179fa21..7cba8777b98 100644 --- a/dag/test/claim/workspace_gateway_operator_witness_test.dag +++ b/dag/test/claim/workspace_gateway_operator_witness_test.dag @@ -1,6 +1,9 @@ module test.claim.workspace_gateway_operator_witness_test -import std.types { Bool } +import std.types { Bool, String } +import gunbc.runner_slot_allocation +import gunbc.workspace_gateway +import gunbc.fleet_known_hosts_anchor import std.process { ExitSuccess, ExitFailure } import gunbc.workspace_gateway_operator { workspace_gateway_refresh_operator_wet } @@ -10,3 +13,11 @@ test fn gateway_refresh_refuses_unbounded_or_noncanonical_durations_before_obser ExitSuccess => false }) } + +test fn workspace_gateway_tailnet_names_have_one_enrolled_host_key() -> Bool { + all(gunbc.runner_slot_allocation.workspace_execution_slot_identities(), slot => + match gunbc.workspace_gateway.workspace_fleet_gateway(slot: slot) { + Absent => false + Present { value: gateway } => count(filter(gunbc.fleet_known_hosts_anchor.fleet_known_hosts_rows(), row => row.endpoint == (gateway.tailnet_name as String))) == 1 + }) +} diff --git a/dag/test/claim/workspace_workflow_dispatch_witness_test.dag b/dag/test/claim/workspace_workflow_dispatch_witness_test.dag index a4829300beb..c1f627b1890 100644 --- a/dag/test/claim/workspace_workflow_dispatch_witness_test.dag +++ b/dag/test/claim/workspace_workflow_dispatch_witness_test.dag @@ -3,7 +3,7 @@ module test.claim.workspace_workflow_dispatch_witness_test import std.types { Bool } import gunbc.fleet_converge_workflow { WorkspaceAllocationPlan, WorkspaceImageBuild, - fleet_converge_mode_scope, fleet_converge_mode_job_id, fleet_converge_shared_job_id, + fleet_converge_dispatch_inputs, fleet_converge_mode_scope, fleet_converge_mode_job_id, fleet_converge_shared_job_id, fleet_converge_mode_release_product, CompilerPairOnly, fleet_converge_mode_fleet_ssh_key_demand, FleetSshKeyNotConsumed, } @@ -36,3 +36,7 @@ test fn workspace_image_build_preserves_ownership_and_archives_only_the_product( && !string_contains(s: paths, pattern: "workspace-tree") && !string_contains(s: paths, pattern: "squashfs") } + +test fn fleet_dispatch_stays_within_github_input_limit() -> Bool { + count(fleet_converge_dispatch_inputs) <= 25 +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 5eb3735acf6..31dcfad3bad 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -14,6 +14,7 @@ collected that refusal; its green workflow conclusion is not a boot receipt. | Boundary | Observed state | Required change | | --- | --- | --- | +| Workflow dispatch | Main declares 28 inputs; GitHub accepts at most 25 | Keep printer selection separate and carry its five operation fields in one `printer_request` JSON input | | Guest image | Builder exists; no image delivery in controller install | Build under fakeroot, archive kernel/rootfs/digest receipt, pin the measured rootfs, verify installed bytes and custody | | Controller | Installed revision `cfb9ff80652fe0a74093f08cade64f59d9695266` | Install this PR through the existing controller installer | | Slot budget | `fabric-cell-srv1-13.slice` has no installed unit file | Converge the modeled execution-cell boundary; a synthesized systemd slice is not proof | @@ -79,3 +80,9 @@ observed. Never record private keys, tokens or signed state-request envelopes. Local validation so far: the image builder produced an ext4 image whose `/root` and `/usr/sbin/sshd` are owned by UID/GID 0, and the fresh compiler parsed the full source tree. Live acceptance is pending. + +The dispatch transport repair preserves the printer's existing domain validation. +Printer callers now supply `printer` and `printer_request`, for example +`{"operation":"observe"}` or +`{"operation":"print","project_path":"/absolute/project.3mf","project_sha256":"","credential_version":"","bed_clear":"true"}`. +No printer action is performed by VM bringup. From 19b12d88d6c0206c3f259cf85b0b664e65000b26 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 03:20:55 +0000 Subject: [PATCH 03/19] Exclude compiler-derived rung-drop roster from deployment candidate --- dag/gunbc/live_deploy/deployed_tree_scope.dag | 3 +++ .../claim/live_deploy/candidate_checkout_witness_test.dag | 6 ++++++ docs/plans/srv1-workspace-vm-bringup.md | 1 + 3 files changed, 10 insertions(+) diff --git a/dag/gunbc/live_deploy/deployed_tree_scope.dag b/dag/gunbc/live_deploy/deployed_tree_scope.dag index a9924e7f206..c3a8525d8bf 100644 --- a/dag/gunbc/live_deploy/deployed_tree_scope.dag +++ b/dag/gunbc/live_deploy/deployed_tree_scope.dag @@ -62,8 +62,11 @@ data deployed_tree_excluded_prefixes: List = [ // recurring_failure_mode/roster.dag). Unlike sg2_scratch_probe, excluding it changes nothing the // host compiles: the host's own first invocation derives the identical file from the sibling rows // the revision DOES name. Exact path, segment-exact, so a sibling row file is never excluded. +// The rung-drop roster has the same compiler-owned derivation. The srv1 workspace deployment +// probe on 2026-10-09 refused itself on that second roster; its sibling authored rows still ship. data deployed_tree_excluded_derived_files: List = [ "dag/gunbc/recurring_failure_mode/roster.dag", + "dag/gunbc/rung_drop/roster.dag", ] // Segment-wise, never a character run: `targetish/x` starts with `target` but is not under it, and treating diff --git a/dag/test/claim/live_deploy/candidate_checkout_witness_test.dag b/dag/test/claim/live_deploy/candidate_checkout_witness_test.dag index 4479a4b9490..8f9cd2d753f 100644 --- a/dag/test/claim/live_deploy/candidate_checkout_witness_test.dag +++ b/dag/test/claim/live_deploy/candidate_checkout_witness_test.dag @@ -345,3 +345,9 @@ test fn the_derived_failure_mode_roster_is_excluded_as_a_file_not_a_prefix() -> && path_is_deployed(path: "dag/gunbc/recurring_failure_mode/empty_capture_read_as_clean_result.dag") && path_is_deployed(path: "dag/gunbc/recurring_failure_mode/roster.dag.bak") } + +test fn the_derived_rung_drop_roster_is_excluded_as_a_file_not_a_prefix() -> Bool { + !path_is_deployed(path: "dag/gunbc/rung_drop/roster.dag") + && path_is_deployed(path: "dag/gunbc/rung_drop/self_host_emit_derisk_claim_unobserved.dag") + && path_is_deployed(path: "dag/gunbc/rung_drop/roster.dag.bak") +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 31dcfad3bad..a88797d238b 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -14,6 +14,7 @@ collected that refusal; its green workflow conclusion is not a boot receipt. | Boundary | Observed state | Required change | | --- | --- | --- | +| Deployment candidate | Compiler startup creates ignored `rung_drop/roster.dag`, which candidate admission rejects | Exclude that exact compiler-derived file alongside the existing failure-mode roster; preserve sibling and secret-file refusals | | Workflow dispatch | Main declares 28 inputs; GitHub accepts at most 25 | Keep printer selection separate and carry its five operation fields in one `printer_request` JSON input | | Guest image | Builder exists; no image delivery in controller install | Build under fakeroot, archive kernel/rootfs/digest receipt, pin the measured rootfs, verify installed bytes and custody | | Controller | Installed revision `cfb9ff80652fe0a74093f08cade64f59d9695266` | Install this PR through the existing controller installer | From 2833a5f1bc3398a03fbe3ad17c335dbbd6be16ac Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 03:28:24 +0000 Subject: [PATCH 04/19] Record VM validation and concrete srv1 deployment blockers --- docs/plans/srv1-workspace-vm-bringup.md | 29 ++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index a88797d238b..b5175a0f25f 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -19,6 +19,7 @@ collected that refusal; its green workflow conclusion is not a boot receipt. | Guest image | Builder exists; no image delivery in controller install | Build under fakeroot, archive kernel/rootfs/digest receipt, pin the measured rootfs, verify installed bytes and custody | | Controller | Installed revision `cfb9ff80652fe0a74093f08cade64f59d9695266` | Install this PR through the existing controller installer | | Slot budget | `fabric-cell-srv1-13.slice` has no installed unit file | Converge the modeled execution-cell boundary; a synthesized systemd slice is not proof | +| Cell inventory | Read-only plan `local-2026-10-09T03:18:35Z` refuses foreign `fabric-cell-zp444.slice` | Establish ownership before retiring it; the empty transient slice has no unit file or remaining journal, and is not silently ignored | | Commissioning | No commissioning or readiness record | Run reviewed commissioning plan/apply and verify its protected transaction | | Protected state service | Running revision `5b96cab749088aa5cec71ce9ad52e1857724d252` predates the authenticated state route | Converge fabric storage through its existing deployment path | | State-key custody | Key exists; operator can read it; `ghrunner` cannot | Preserve custody; exact installed helpers reconcile admitted intent and read the selected allocation | @@ -78,9 +79,31 @@ same-slot reuse with a new incarnation, expiry cleanup, and final released/ready state. Record run IDs, revisions, allocation IDs and generations here as they are observed. Never record private keys, tokens or signed state-request envelopes. -Local validation so far: the image builder produced an ext4 image whose `/root` -and `/usr/sbin/sshd` are owned by UID/GID 0, and the fresh compiler parsed the full -source tree. Live acceptance is pending. +Local validation: 39 focused witnesses passed across allocation dispatch, lifecycle, +launch directives, host offers, owner policy, gateway refresh, workflow dispatch and +deployment candidate admission. The image builder produced an ext4 image whose +`/root` and `/usr/sbin/sshd` are owned by UID/GID 0. The fresh compiler parsed the +full source tree, and full generated-artifact regeneration completed successfully. +The combined workflow witness process needed a 24 GiB memory cap; its earlier +16 GiB run was killed by the cgroup limit, then passed with the larger cap. + +[Image build 37878358778](https://github.com/gunb-ai/gunbc/actions/runs/37878358778) +was accepted at revision `521085ab254290029f8dfbaa312a218f84a6be7b` and is waiting +for runner capacity. The current rootfs pin is the measured local build and must +be updated to that archived build's digest before controller installation. No +service deployment, commissioning, allocation or VM mutation has been performed +for this acceptance run yet. + +The read-only deployment probe admitted clean candidate +`19b12d88d6c0206c3f259cf85b0b664e65000b26` after the derived-roster repair. Its +release-member plan is: install executable +`sha256:5cbf98426843aebd1fad3c2f9cc52c6ba04ba6fbb71f6bbde1ea40eff8391543`, +publish that source revision, write `gunbc-roadmap.service`, reload systemd and +restart the roadmap service. The existing deployment additionally reinstalls its +fabric-storage, approval-broker, tailnet-door and timer members, which do not yet +have differential member identities. Consequently this prerequisite is a shared +production deployment with service interruption, not a slot-only mutation. +Operator approval is pending before applying that broader deployment. The dispatch transport repair preserves the printer's existing domain validation. Printer callers now supply `printer` and `printer_request`, for example From 5504b52cd0001115afcfb47a98e98b16c44e199c Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 04:04:54 +0000 Subject: [PATCH 05/19] Repair workspace image diagnostics and refused-deploy readiness --- .github/workflows/fleet-converge.yml | 22 +++++++++++- dag/gunbc/fleet/fleet_release_bins_key.dag | 8 +++-- dag/gunbc/live_deploy/apply.dag | 16 +++++---- ...eadiness_polled_after_refused_mutation.dag | 14 ++++++++ .../receipt_consumer_without_producer.dag | 1 + docs/plans/srv1-workspace-vm-bringup.md | 36 ++++++++++++++----- 6 files changed, 80 insertions(+), 17 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/readiness_polled_after_refused_mutation.dag diff --git a/.github/workflows/fleet-converge.yml b/.github/workflows/fleet-converge.yml index df414398708..782b58b26e7 100644 --- a/.github/workflows/fleet-converge.yml +++ b/.github/workflows/fleet-converge.yml @@ -518,7 +518,27 @@ jobs: id: release_build run: | if [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_allocation_plan' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_image_build' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'approval_device_enrolment_code_issue' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_source_pack' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_measure' ] || [ "${FLEET_CONVERGE_MODE:-}" = 'workspace_checkpoint_restore' ]; then - cargo build --release -p v1-compiler --bin claim_executor --bin gunbc + GUNBC_FLOOR_LOG='gunbc-floor-cmd.log' + 'set' '+e' + 'set' '-o' 'pipefail' + ('sh' '-e' '-c' 'cargo build --release -p v1-compiler --bin claim_executor --bin gunbc') 2>&1 | 'tee' "$GUNBC_FLOOR_LOG" + GUNBC_FLOOR_EXIT="$?" + GUNBC_FLOOR_RECEIPT='gunbc-floor-outcome.txt' + GUNBC_FLOOR_CLASS='structural' + GUNBC_FLOOR_SIGNATURE='' + if '[' "$GUNBC_FLOOR_EXIT" '-eq' '126' ']'; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='CommandInvokedCannotExecute'; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' "$GUNBC_FLOOR_EXIT" '-eq' '127' ']'; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='CommandNotFound'; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' "$GUNBC_FLOOR_EXIT" '-eq' '0' ']'; then GUNBC_FLOOR_CLASS='none'; GUNBC_FLOOR_SIGNATURE=''; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' '-f' "$GUNBC_FLOOR_LOG" ']' && 'grep' '-q' 'MemoryStallRefusedPageThrash' "$GUNBC_FLOOR_LOG"; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='MemoryStallRefusedPageThrash'; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' '-f' "$GUNBC_FLOOR_LOG" ']' && 'grep' '-q' 'The runner has received a shutdown signal' "$GUNBC_FLOOR_LOG"; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='RunnerLost'; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' '-f' "$GUNBC_FLOOR_LOG" ']' && 'grep' '-q' 'sccache: error: failed to execute compile' "$GUNBC_FLOOR_LOG"; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='SccacheFailedToExecuteCompile'; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' '-f' "$GUNBC_FLOOR_LOG" ']' && 'grep' '-q' '(exit status: 254)' "$GUNBC_FLOOR_LOG"; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='SccacheRustcWrapperExit254'; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' '-f' "$GUNBC_FLOOR_LOG" ']' && 'grep' '-q' 'sccache: encountered fatal error' "$GUNBC_FLOOR_LOG"; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='SccacheFatalError'; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' '-f' "$GUNBC_FLOOR_LOG" ']' && 'grep' '-q' 'failed to spawn' "$GUNBC_FLOOR_LOG"; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='ProcessSpawnFailure'; fi + if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' && '[' '-f' "$GUNBC_FLOOR_LOG" ']' && 'grep' '-q' 'Resource temporarily unavailable' "$GUNBC_FLOOR_LOG"; then GUNBC_FLOOR_CLASS='infra'; GUNBC_FLOOR_SIGNATURE='ResourceTemporarilyUnavailable'; fi + if (! '[' '-f' "$GUNBC_FLOOR_RECEIPT" ']') || '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']' || ('[' "$GUNBC_FLOOR_CLASS" '=' 'infra' ']' && (! ('[' '-f' "$GUNBC_FLOOR_RECEIPT" ']' && 'grep' '-q' 'class=structural' "$GUNBC_FLOOR_RECEIPT"))) || ('[' "$GUNBC_FLOOR_CLASS" '=' 'none' ']' && (! ('[' '-f' "$GUNBC_FLOOR_RECEIPT" ']' && 'grep' '-q' 'class=structural' "$GUNBC_FLOOR_RECEIPT")) && (! ('[' '-f' "$GUNBC_FLOOR_RECEIPT" ']' && 'grep' '-q' 'class=infra' "$GUNBC_FLOOR_RECEIPT"))); then 'printf' 'class=%s\nsignature=%s\nexit=%s\n' "$GUNBC_FLOOR_CLASS" "$GUNBC_FLOOR_SIGNATURE" "$GUNBC_FLOOR_EXIT" > "$GUNBC_FLOOR_RECEIPT"; if '[' "$GUNBC_FLOOR_CLASS" '=' 'infra' ']'; then 'echo' '::error title=environment::floor_class='"$GUNBC_FLOOR_CLASS"' signature='"$GUNBC_FLOOR_SIGNATURE"' exit='"$GUNBC_FLOOR_EXIT"'; this is not a verdict about the diff. Attempt receipt: '"$GUNBC_FLOOR_RECEIPT"; fi; if '[' "$GUNBC_FLOOR_CLASS" '=' 'structural' ']'; then 'echo' '::error title=subject::floor_class='"$GUNBC_FLOOR_CLASS"' exit='"$GUNBC_FLOOR_EXIT"'; read the step log for the subject defect'; fi; fi + 'exit' "$GUNBC_FLOOR_EXIT" + else ROOT=$('git' 'rev-parse' '--show-toplevel') set -o pipefail diff --git a/dag/gunbc/fleet/fleet_release_bins_key.dag b/dag/gunbc/fleet/fleet_release_bins_key.dag index a85e69d4457..e45acd89623 100644 --- a/dag/gunbc/fleet/fleet_release_bins_key.dag +++ b/dag/gunbc/fleet/fleet_release_bins_key.dag @@ -45,7 +45,7 @@ import extdeps.git.object_store { git_object_type_wire_name } import gunbc.ci_release_bins { ci_floor_required_artifact_names, ci_floor_invoked_bins } import gunbc.repo_self_build { repo_self_build_command } import gunbc.ci_spec { ci_release_build_line, ci_release_bins_archive_name } -import gunbc.ci_failure_class { floor_attempt_command_log_filename } +import gunbc.ci_failure_class { floor_attempt_command_log_filename, wrap_command_with_floor_attempt_receipt } // THE RELEASE-BINS PACK IS A CONTENT-ADDRESSED MATERIALIZATION, KEYED BEFORE IT IS BUILT (operator // ruling 2026-09-27, route B chosen by eager-owl-205: the fleet-converge build job is the keyed @@ -1381,7 +1381,11 @@ fn release_bins_consumer_script_dispatch(pair_modes: List) -> String { } fn release_bins_build_script_dispatch(pair_modes: List, fleet_build: String) -> String { - release_bins_mode_dispatch(pair_modes: pair_modes, pair_text: compiler_pair_release_product().build_line, fleet_text: fleet_build) + // The pair's pack includes the build diagnostics. Use the floor's existing capture wrapper, + // which writes that log and preserves a failed compiler exit through tee. + release_bins_mode_dispatch(pair_modes: pair_modes, + pair_text: wrap_command_with_floor_attempt_receipt(command: compiler_pair_release_product().build_line), + fleet_text: fleet_build) } // THE EMITTED SCRIPTS, and the one admission the workflow generation consumes: if the fold refuses any diff --git a/dag/gunbc/live_deploy/apply.dag b/dag/gunbc/live_deploy/apply.dag index 0abd11d7880..ed21ef0d768 100644 --- a/dag/gunbc/live_deploy/apply.dag +++ b/dag/gunbc/live_deploy/apply.dag @@ -557,12 +557,16 @@ fn live_deploy_apply_via_transport( transport: transport, access: access, ) - let readiness = live_deploy_poll_until_service_ready_for_deploy( - spec: spec, - candidate: candidate, - transport: transport, - access: access, - ) + // A refused mutation cannot produce this candidate. Skip startup polling, but keep the + // cutover gate below: it must still recover an interrupted route transaction. + let readiness = if reconciliation_converged(r: mutation) { + live_deploy_poll_until_service_ready_for_deploy( + spec: spec, + candidate: candidate, + transport: transport, + access: access, + ) + } else { tailnet_door_cutover_unpermitted_standing() } let cutover = live_deploy_tailnet_door_cutover_gate( spec: spec, candidate: candidate, transport: transport, access: access, readiness: tailnet_door_cutover_door_standing(mutation: mutation, readiness: readiness), diff --git a/dag/gunbc/recurring_failure_mode/readiness_polled_after_refused_mutation.dag b/dag/gunbc/recurring_failure_mode/readiness_polled_after_refused_mutation.dag new file mode 100644 index 00000000000..4dbe99052d5 --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/readiness_polled_after_refused_mutation.dag @@ -0,0 +1,14 @@ +module gunbc.recurring_failure_mode.readiness_polled_after_refused_mutation + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data readiness_polled_after_refused_mutation: RecurringFailureMode = RecurringFailureMode { + identity: "readiness_polled_after_refused_mutation" as NonEmptyStr, + receipts: [ + "INVALID STATE: a deployment polls for a candidate after its mutation already refused, withholding the original error until the startup bound expires. An old service answering health probes is not evidence that the failed installation needs more startup time.", + "OBSERVED 2026-10-09 during srv1 workspace bringup, PR #13607, candidate 19b12d88d6c0206c3f259cf85b0b664e65000b26: the operator-local deploy installed its generated ghrunner sudoers, then refused the exact tailscale grant probe under the operator account. It nevertheless performed 720 readiness waits against the previous release before returning the refusal. Independent grant-list readback admits the same exact path for ghrunner and refuses it for the operator; the retry uses the existing deployment-account entry without widening the operator grant roster.", + "REPAIR: live_deploy_apply_via_transport polls only after a converged mutation. A refused mutation supplies an unpermitted door standing and still reaches the cutover recovery gate: an earlier interrupted route transaction may require rollback, even when this mutation failed. Skipping the entire remainder would remove that recovery. CURRENT RUNG: mechanically prevented at this caller; the existing readiness and cutover refusal controls remain required.", + ], + evidence: [], +} diff --git a/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag b/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag index 964d3116d25..c317bd2ab92 100644 --- a/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag +++ b/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag @@ -6,6 +6,7 @@ import gunbc.recurring_failure_mode { RecurringFailureMode } data receipt_consumer_without_producer: RecurringFailureMode = RecurringFailureMode { identity: "receipt_consumer_without_producer" as NonEmptyStr, receipts: [ + "2026-10-09 srv1 workspace image run 37878358778, PR #13607: the compiler pair built successfully, then release-bins packing refused BuildDiagnosticsMissing because the product requires gunbc-floor-cmd.log but fleet's pair build dispatch ran the raw cargo line without the floor's capture wrapper. The pack's refusal was correct. Reusing wrap_command_with_floor_attempt_receipt at that producer captures the required diagnostics and preserves compiler failure through tee; no empty log is synthesized and no pack member is dropped.", "2026-10-09 srv1 workspace bringup, PR #13607: allocation offer admission and the controller require a gateway forwarding observation no older than 60 seconds, but workspace_refresh_gateway_wet had no production caller. The observer existed and its decoder was tested, while every real offer necessarily refused an absent or stale receipt. A bounded foreground operator command now calls the existing observer and protected publication repeatedly using the enrolled agent and host keys; no scheduler or private-key copy is introduced. Acceptance still requires the live forwarding receipt and authenticated guest SSH, not just the command returning success.", "PR #12421 review 5331195485: gunbc_ci_namecheap_observe_invoke supplied receipt_rel as though it captured execution output, but gunbc_invoke_receipt_steps only read an existing file. A successful observer therefore failed the shell step and suppressed its artifact upload. Invalid state: an emitted receipt read with no producer. Harm: successful observation becomes workflow failure. Discriminating facts: execute emitted shell in a clean root with only the declared output; a planted unmatched read must fail. Mitigation: writer, console display and uploader derive the path from one declaration. Ceiling: workflow construction rejects a receipt consumer without a producer edge; next trigger: that consumed workflow admission relation.", ], diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index b5175a0f25f..eb0157c4010 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -79,20 +79,32 @@ same-slot reuse with a new incarnation, expiry cleanup, and final released/ready state. Record run IDs, revisions, allocation IDs and generations here as they are observed. Never record private keys, tokens or signed state-request envelopes. -Local validation: 39 focused witnesses passed across allocation dispatch, lifecycle, +Local validation: 44 focused witnesses passed across allocation dispatch, lifecycle, launch directives, host offers, owner policy, gateway refresh, workflow dispatch and -deployment candidate admission. The image builder produced an ext4 image whose +deployment candidate admission, including three deployment-readiness refusal controls and two cutover recovery/refusal controls. The generated compiler-pair build step was also executed with controlled compiler outcomes: both output streams were retained, and exit codes 0 and 37 were preserved. The image builder produced an ext4 image whose `/root` and `/usr/sbin/sshd` are owned by UID/GID 0. The fresh compiler parsed the full source tree, and full generated-artifact regeneration completed successfully. The combined workflow witness process needed a 24 GiB memory cap; its earlier 16 GiB run was killed by the cgroup limit, then passed with the larger cap. [Image build 37878358778](https://github.com/gunb-ai/gunbc/actions/runs/37878358778) -was accepted at revision `521085ab254290029f8dfbaa312a218f84a6be7b` and is waiting -for runner capacity. The current rootfs pin is the measured local build and must -be updated to that archived build's digest before controller installation. No -service deployment, commissioning, allocation or VM mutation has been performed -for this acceptance run yet. +at revision `521085ab254290029f8dfbaa312a218f84a6be7b` built its compiler successfully, +but packaging refused `BuildDiagnosticsMissing`: the compiler-pair build had not +produced the log required by its pack. The build dispatch now uses the existing +floor log-capture wrapper, preserving the compiler's exit status. A new image run +is required. The current rootfs pin remains the measured local build and must be +updated to the archived build's digest before controller installation. + +The operator identified `fabric-cell-zp444.slice` as a disposable test slice and +authorized retirement. Immediately before stopping it, readback confirmed no unit +file, drop-ins, child units or processes. Its unit and cgroup are now absent. +Execution-cell plan `local-2026-10-09T03:37:32Z`, hash `76783579d393ed3d`, +advanced generation 18 to 19. Apply `local-2026-10-09T03:47:17Z` completed +`fully_applied`, receipt `34c5252f521bf5b3`. Independent systemd readback confirms +active `fabric-cell-srv1-13.slice`, persistent limits of 28 GiB for MemoryMax and +MemoryHigh, no swap, TasksMax 16384, and the modeled CPU settings. The cell and +attempt directories have the modeled custody. Commissioning, allocation and VM +boot remain unproven. The read-only deployment probe admitted clean candidate `19b12d88d6c0206c3f259cf85b0b664e65000b26` after the derived-roster repair. Its @@ -103,7 +115,15 @@ restart the roadmap service. The existing deployment additionally reinstalls its fabric-storage, approval-broker, tailnet-door and timer members, which do not yet have differential member identities. Consequently this prerequisite is a shared production deployment with service interruption, not a slot-only mutation. -Operator approval is pending before applying that broader deployment. +The operator approved this shared deployment, including temporary interruption. +The first operator-local attempt installed the generated `ghrunner` sudoers but +refused its exact `tailscale` grant probe under the operator account before other +mutations. The operator's broad sudo grant does not satisfy this exact grant-list +probe; the same probe admits the existing `ghrunner` grant. The retry uses the +existing `live_deploy_apply_srv1_wet` entry as `ghrunner`, without expanding the +operator grant roster. The failed attempt also exposed 720 unnecessary readiness +waits after refusal. Readiness polling now requires a converged mutation; cutover +recovery still runs so interrupted route changes can be rolled back. The dispatch transport repair preserves the printer's existing domain validation. Printer callers now supply `printer` and `printer_request`, for example From 95037360ab1040b9de4efe4671bb53cad473cb34 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 04:30:20 +0000 Subject: [PATCH 06/19] Pin the verified archived workspace guest image --- dag/gunbc/workspace_artifact_policy.dag | 2 +- docs/plans/srv1-workspace-vm-bringup.md | 38 +++++++++++++++++++++++-- 2 files changed, 36 insertions(+), 4 deletions(-) diff --git a/dag/gunbc/workspace_artifact_policy.dag b/dag/gunbc/workspace_artifact_policy.dag index ba286b8bedb..d8a9f000d6d 100644 --- a/dag/gunbc/workspace_artifact_policy.dag +++ b/dag/gunbc/workspace_artifact_policy.dag @@ -5,5 +5,5 @@ import std.types { NonEmptyStr } // The pinned workspace guest image the commissioning fleet path installs and admits. It is the // output digest of the pinned-base workspace image builder (gunbc.workspace_guest_image). fn workspace_demo_image_identity() -> NonEmptyStr { - "a64c13977ed6e493e75a6edc3a8b42eb713b1d2de2c3e43759b33550e31af553" + "d15a55780a707ec832a09e3599306254b8d7c428ff25ff621b427b826eeb0b27" } diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index eb0157c4010..5c2cb145684 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -91,9 +91,13 @@ The combined workflow witness process needed a 24 GiB memory cap; its earlier at revision `521085ab254290029f8dfbaa312a218f84a6be7b` built its compiler successfully, but packaging refused `BuildDiagnosticsMissing`: the compiler-pair build had not produced the log required by its pack. The build dispatch now uses the existing -floor log-capture wrapper, preserving the compiler's exit status. A new image run -is required. The current rootfs pin remains the measured local build and must be -updated to the archived build's digest before controller installation. +floor log-capture wrapper, preserving the compiler's exit status. [Retry 37882239644](https://github.com/gunb-ai/gunbc/actions/runs/37882239644) +at revision `5504b52cd0001115afcfb47a98e98b16c44e199c` succeeded. The downloaded +artifact's rootfs SHA-256 matches its receipt: +`d15a55780a707ec832a09e3599306254b8d7c428ff25ff621b427b826eeb0b27`. +That archived digest is now the reviewed image pin. The kernel matches +`cb1291c66bca75bc11cb9c8357fcef9965bb1786dffcb42a60923c3e0e49f319`; +independent ext4 inspection confirms `/root` and `/usr/sbin/sshd` belong to UID/GID 0. The operator identified `fabric-cell-zp444.slice` as a disposable test slice and authorized retirement. Immediately before stopping it, readback confirmed no unit @@ -130,3 +134,31 @@ Printer callers now supply `printer` and `printer_request`, for example `{"operation":"observe"}` or `{"operation":"print","project_path":"/absolute/project.3mf","project_sha256":"","credential_version":"","bed_clear":"true"}`. No printer action is performed by VM bringup. + +The approved deployment's tree sync and independent repository readback completed +at revision `19b12d88d6c0206c3f259cf85b0b664e65000b26`. Dashboard, approval broker, +tailnet door and fabric storage restarted between 04:11:48 and 04:12:04 UTC; +dashboard readiness is still being checked. At 04:21 UTC the authenticated operator read returned `WorkspaceOperatorObserveRefused: request absent for operator`, proving the protected owner ledger is readable without creating an allocation. The preceding probe finished 19 seconds before the new storage listener opened and is not counted as acceptance. +The nested deployment/readback interpreters peaked above the original 24 GiB +scope bound, so that operation's cap was raised to 36 GiB with swap disabled. + +The enrolled operator key is the MacBook key, fingerprint +`SHA256:KEMuhG7TCEjoaF2Q8Xn/4jJd+5uZUggzycUYnLXQojI`. It is already authorized +on srv1. The local srv1/srv2 keys differ and no agent was selected. Gateway and +guest SSH acceptance therefore require the operator's forwarded agent; its socket +has been requested. No enrollment or authorized-key policy is being widened. + +The deployment installed its members but returned a cutover refusal: the existing +`ghrunner` deployment entry cannot read the protected-state credential. Its final +cutover gate therefore could not read the journal and performed no route effect. +The dashboard's loopback `/healthz` independently reports revision `19b12d88d6c`; +the existing tailnet root route still targets that loopback backend. The new +parallel tailnet-door unit also failed binding `/opt/gunbc/tailnet-door.sock`: +its service user cannot create entries in root-owned `/opt/gunbc`. These are +shared-deployment gaps, not permission to distribute the state key or re-own the +installation root. Full shared-deployment convergence is not claimed. + +During cold startup, the shared dashboard slice exceeded its 49 GiB memory-high +threshold (53 GiB hard cap) and spent about 60% of wall time stalled. Publication +and belt timers/helpers were temporarily stopped within the approved interruption +window to let the services finish startup. Their restoration remains required. From a85cc481af5b07dc89f270925bb55bc6f025c8cd Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 05:11:09 +0000 Subject: [PATCH 07/19] Support workspace SSH directly from the host with a dedicated guest key --- dag/gunbc/fleet/fleet_release_bins_key.dag | 4 +- dag/gunbc/live_deploy/apply.dag | 4 +- .../receipt_consumer_without_producer.dag | 2 +- .../runner/runner_microvm_slot_controller.dag | 42 ++++++---- dag/gunbc/runner/workspace_microvm_access.dag | 51 +++++++------ dag/gunbc/workspace_allocation_admission.dag | 9 ++- dag/gunbc/workspace_allocation_policy.dag | 43 ++++++++--- dag/gunbc/workspace_gateway_operator.dag | 76 ------------------- dag/gunbc/workspace_host_offer_observe.dag | 26 +++++-- dag/gunbc/workspace_offer_producer.dag | 56 ++++++++------ dag/gunbc/workspace_ssh_access.dag | 65 ++++++++++++++++ .../runner/workspace_microvm_witness_test.dag | 48 +++++++++++- .../workspace_owner_access_witness_test.dag | 53 ++++++++----- ...orkspace_gateway_operator_witness_test.dag | 23 ------ .../workspace_reconcile_once_witness_test.dag | 4 +- .../workspace_ssh_access_witness_test.dag | 56 ++++++++++++++ docs/plans/srv1-workspace-vm-bringup.md | 51 +++++++++---- 17 files changed, 388 insertions(+), 225 deletions(-) delete mode 100644 dag/gunbc/workspace_gateway_operator.dag create mode 100644 dag/gunbc/workspace_ssh_access.dag delete mode 100644 dag/test/claim/workspace_gateway_operator_witness_test.dag create mode 100644 dag/test/claim/workspace_ssh_access_witness_test.dag diff --git a/dag/gunbc/fleet/fleet_release_bins_key.dag b/dag/gunbc/fleet/fleet_release_bins_key.dag index e45acd89623..d59d10e3bdb 100644 --- a/dag/gunbc/fleet/fleet_release_bins_key.dag +++ b/dag/gunbc/fleet/fleet_release_bins_key.dag @@ -1380,9 +1380,9 @@ fn release_bins_consumer_script_dispatch(pair_modes: List) -> String { release_bins_mode_dispatch(pair_modes: pair_modes, pair_text: release_bins_consumer_key_check_script_for(product: compiler_pair_release_product()), fleet_text: release_bins_consumer_key_check_script()) } +// The pair's pack includes the build diagnostics. Use the floor's existing capture wrapper, +// which writes that log and preserves a failed compiler exit through tee. fn release_bins_build_script_dispatch(pair_modes: List, fleet_build: String) -> String { - // The pair's pack includes the build diagnostics. Use the floor's existing capture wrapper, - // which writes that log and preserves a failed compiler exit through tee. release_bins_mode_dispatch(pair_modes: pair_modes, pair_text: wrap_command_with_floor_attempt_receipt(command: compiler_pair_release_product().build_line), fleet_text: fleet_build) diff --git a/dag/gunbc/live_deploy/apply.dag b/dag/gunbc/live_deploy/apply.dag index ed21ef0d768..37d5f01bbf7 100644 --- a/dag/gunbc/live_deploy/apply.dag +++ b/dag/gunbc/live_deploy/apply.dag @@ -545,6 +545,8 @@ fn live_deploy_apply_identity_readback_gate( } } +// A refused mutation cannot produce this candidate. Skip startup polling, but keep the +// cutover gate below: it must still recover an interrupted route transaction. fn live_deploy_apply_via_transport( spec: DeploymentSpec, candidate: CandidateRelease, @@ -557,8 +559,6 @@ fn live_deploy_apply_via_transport( transport: transport, access: access, ) - // A refused mutation cannot produce this candidate. Skip startup polling, but keep the - // cutover gate below: it must still recover an interrupted route transaction. let readiness = if reconciliation_converged(r: mutation) { live_deploy_poll_until_service_ready_for_deploy( spec: spec, diff --git a/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag b/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag index c317bd2ab92..f3a1eaf186b 100644 --- a/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag +++ b/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag @@ -7,7 +7,7 @@ data receipt_consumer_without_producer: RecurringFailureMode = RecurringFailureM identity: "receipt_consumer_without_producer" as NonEmptyStr, receipts: [ "2026-10-09 srv1 workspace image run 37878358778, PR #13607: the compiler pair built successfully, then release-bins packing refused BuildDiagnosticsMissing because the product requires gunbc-floor-cmd.log but fleet's pair build dispatch ran the raw cargo line without the floor's capture wrapper. The pack's refusal was correct. Reusing wrap_command_with_floor_attempt_receipt at that producer captures the required diagnostics and preserves compiler failure through tee; no empty log is synthesized and no pack member is dropped.", - "2026-10-09 srv1 workspace bringup, PR #13607: allocation offer admission and the controller require a gateway forwarding observation no older than 60 seconds, but workspace_refresh_gateway_wet had no production caller. The observer existed and its decoder was tested, while every real offer necessarily refused an absent or stale receipt. A bounded foreground operator command now calls the existing observer and protected publication repeatedly using the enrolled agent and host keys; no scheduler or private-key copy is introduced. Acceptance still requires the live forwarding receipt and authenticated guest SSH, not just the command returning success.", + "2026-10-09 srv1 workspace bringup, PR #13607: allocation offer admission and the controller require a gateway forwarding observation no older than 60 seconds, but workspace_refresh_gateway_wet had no production caller. The observer existed and its decoder was tested, while every real offer necessarily refused an absent or stale receipt. A bounded foreground operator refresher was initially added, then removed when the operator selected direct SSH from srv1. That route does not exercise gateway forwarding and must not require its receipt: workspace_ssh_access now carries distinct host-local and gateway routes, and each consumer requires only the evidence for its enrolled route. Gateway clients retain their forwarding-proof requirement. Host-local acceptance still requires the invocation-bound guest host key and authenticated guest SSH; a fresh host observation alone is not a login receipt.", "PR #12421 review 5331195485: gunbc_ci_namecheap_observe_invoke supplied receipt_rel as though it captured execution output, but gunbc_invoke_receipt_steps only read an existing file. A successful observer therefore failed the shell step and suppressed its artifact upload. Invalid state: an emitted receipt read with no producer. Harm: successful observation becomes workflow failure. Discriminating facts: execute emitted shell in a clean root with only the declared output; a planted unmatched read must fail. Mitigation: writer, console display and uploader derive the path from one declaration. Ceiling: workflow construction rejects a receipt consumer without a producer edge; next trigger: that consumed workflow admission relation.", ], evidence: [], diff --git a/dag/gunbc/runner/runner_microvm_slot_controller.dag b/dag/gunbc/runner/runner_microvm_slot_controller.dag index 3dbce2f7117..982c75b32f2 100644 --- a/dag/gunbc/runner/runner_microvm_slot_controller.dag +++ b/dag/gunbc/runner/runner_microvm_slot_controller.dag @@ -36,14 +36,16 @@ import gunbc.workspace_allocation_acquisition { workspace_apply_recorded_plan } import gunbc.workspace_allocation_lifecycle { workspace_complete_teardown, workspace_recover_release } import gunbc.runner_attempt_launch { LaunchWithdrawn } import gunbc.workspace_microvm { workspace_retirement_identity } -import gunbc.workspace_allocation_policy { workspace_installed_policy } +import gunbc.workspace_allocation_policy { workspace_installed_policy, workspace_enrolled_ssh_access } +import gunbc.workspace_ssh_access { WorkspaceSshFromHost, WorkspaceSshThroughGateway, WorkspaceDirectSsh, WorkspaceGatewaySsh } +import gunbc.principal_projection { PrincipalRef } import gunbc.workspace_allocation_admission { workspace_owner_enrollment, WorkspacePolicyAdmitted, WorkspacePolicyRefused } import gunbc.workspace_allocation { WorkspaceReservation } import gunbc.workspace_allocation_store { WorkspaceAllocation, WorkspaceReleased, WorkspaceWanted, workspace_read_owner, WorkspaceStoreReadOk, WorkspaceStoreUnreadable } import gunbc.workspace_reservation_record { workspace_reservation_plan_reservation } import gunbc.workspace_guest_image { workspace_installed_image, WorkspaceImageVerified, WorkspaceImageRefused } import gunbc.workspace_microvm { bind_workspace_launch, workspace_owner_public_key, WorkspacePublicKeyAdmitted, WorkspacePublicKeyNotAdmitted, WorkspaceLaunchBound, WorkspaceLaunchRefused } -import gunbc.workspace_gateway { WorkspaceGateway, workspace_fleet_gateway } +import gunbc.workspace_gateway { workspace_fleet_gateway } import gunbc.runner_microvm_attempt { microvm_attempt, MicroVmAttemptAccepted, MicroVmAttemptRefused, attempt_segment_refusal_text } import gunbc.fabric_required_build_cell { observe_required_build_cell_for_identity } import gunbc.fabric_control_plane { fabric_cell_allocation_key } @@ -723,7 +725,6 @@ fn run_workspace_slot( bins: LocusBinaries, invocation_id: NonEmptyStr, allocation_store: FabricStorageBinding, - gateway: WorkspaceGateway, ) -> ControllerOutcome { let identity = workspace_attempt_identity(binding: binding, invocation_id: invocation_id, firecracker_binary: bins.firecracker) let plan = workspace_launch_plan(binding: binding, ground: observe_attempt_ground(attempt: binding.attempt), jailer_binary: bins.jailer, firecracker_binary: bins.firecracker) @@ -734,7 +735,7 @@ fn run_workspace_slot( store: allocation_store, owner: binding.reservation.request.owner, idempotency_key: binding.reservation.request.idempotency_key, identity: live, launch: launch, - observe_connection: (observed, started) => workspace_observe_admitted_connection(store: allocation_store, identity: observed, launch: started, gateway: gateway), + observe_connection: (observed, started) => workspace_observe_admitted_connection(store: allocation_store, identity: observed, launch: started, owner: binding.reservation.request.owner), ) let measured = workspace_capture_runtime_metrics(store: allocation_store, reservation: binding.reservation, identity: live, launch: launch) if measured { terminal } else { terminal } @@ -742,13 +743,25 @@ fn run_workspace_slot( ) } -// Runtime reads the operator-context forwarding observation; it never borrows the operator's key. -fn workspace_observe_admitted_connection(store: FabricStorageBinding, identity: AttemptIdentity, launch: AttemptLaunch, gateway: WorkspaceGateway) -> WorkspaceConnectionReading { - match clock_now_probed_at() { - Absent => WorkspaceConnectionUnavailable { reason: "gateway admission clock is unreadable" } - Present { value: now } => match workspace_current_gateway_connection(store: store, slot: identity.attempt.slot, now: now) { - WorkspaceGatewayUnavailable { reason } => WorkspaceConnectionUnavailable { reason: reason as String } - WorkspaceGatewayReady { known_host } => workspace_observe_connection(identity: identity, launch: launch, gateway: gateway, gateway_known_host: known_host) +// The controller has already bound this host and allocation. A host-local client needs the +// invocation-bound guest key exchange; only a gateway client additionally needs forwarding proof. +fn workspace_observe_admitted_connection(store: FabricStorageBinding, identity: AttemptIdentity, launch: AttemptLaunch, owner: PrincipalRef) -> WorkspaceConnectionReading { + match workspace_enrolled_ssh_access(owner: owner) { + Absent => WorkspaceConnectionUnavailable { reason: "workspace owner has no admitted SSH access policy" } + Present { value: access } => match access { + WorkspaceSshFromHost { identity_file } => workspace_observe_connection(identity: identity, launch: launch, + route: WorkspaceDirectSsh { slot: identity.attempt.slot, identity_file: identity_file }) + WorkspaceSshThroughGateway => match workspace_fleet_gateway(slot: identity.attempt.slot) { + Absent => WorkspaceConnectionUnavailable { reason: "fleet tailnet gateway is unconfigured" } + Present { value: gateway } => match clock_now_probed_at() { + Absent => WorkspaceConnectionUnavailable { reason: "gateway admission clock is unreadable" } + Present { value: now } => match workspace_current_gateway_connection(store: store, slot: identity.attempt.slot, now: now) { + WorkspaceGatewayUnavailable { reason } => WorkspaceConnectionUnavailable { reason: reason as String } + WorkspaceGatewayReady { known_host } => workspace_observe_connection(identity: identity, launch: launch, + route: WorkspaceGatewaySsh { gateway: gateway, known_host: known_host }) + } + } + } } } } @@ -962,12 +975,9 @@ fn workspace_slot_launch_under_exclusion(reservation: WorkspaceReservation, live ClockUnixMillisObserved { millis } => match bind_workspace_launch(reservation: reservation, snapshot: live, attempt: attempt, image: image, public_key: owner_key, now_unix: epoch_secs_of_millis(millis: millis)) { WorkspaceLaunchRefused { reason } => workspace_slot_refused(reason: reason) - WorkspaceLaunchBound { binding } => match workspace_fleet_gateway(slot: reservation.slot) { - Absent => workspace_slot_refused(reason: "fleet tailnet gateway is unconfigured") - Present { value: gateway } => workspace_settle_controller_outcome(reservation: reservation, store: store, + WorkspaceLaunchBound { binding } => workspace_settle_controller_outcome(reservation: reservation, store: store, outcome: run_workspace_slot(binding: binding, subject: subject, bins: bins, - invocation_id: invocation_id, allocation_store: store, gateway: gateway)) - } + invocation_id: invocation_id, allocation_store: store)) } ClockUnixMillisRefused { detail: _ } => workspace_slot_refused(reason: "workspace launch clock is unreadable") } diff --git a/dag/gunbc/runner/workspace_microvm_access.dag b/dag/gunbc/runner/workspace_microvm_access.dag index ff7a3e17192..960d819954e 100644 --- a/dag/gunbc/runner/workspace_microvm_access.dag +++ b/dag/gunbc/runner/workspace_microvm_access.dag @@ -1,7 +1,7 @@ module gunbc.workspace_microvm_access import extdeps.systemd.journalctl -import gunbc.workspace_gateway { WorkspaceGateway } +import gunbc.workspace_ssh_access { WorkspaceConnectionRoute, WorkspaceDirectSsh, WorkspaceGatewaySsh, workspace_connection_route_refusal } import std.types { String, NonEmptyStr, Bool, List, Int } import std.algebra { trim } @@ -12,7 +12,6 @@ import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent, ProcessRe import extdeps.ssh.openssh_client_commands { ssh_keyscan_command } import extdeps.network.ipv4 { render_ipv4_address } import gunbc.command_runner { run_shell_command_capture } -import gunbc.build_cache_instance { RunnerSlotIdentity, runner_slot_identity_equal } import gunbc.runner_microvm_network { runner_slot_network } import gunbc.runner_microvm_lifecycle { AttemptIdentity, in_flight_attempt_of_identity } import gunbc.runner_microvm_cell_readiness_store { encode_attempt } @@ -25,9 +24,7 @@ import gunbc.runner_microvm_lifecycle_realize { type WorkspaceConnection sole_constructor { identity: AttemptIdentity systemd_invocation: NonEmptyStr - host_account: NonEmptyStr - host_tailnet: NonEmptyStr - gateway_known_host: NonEmptyStr + route: WorkspaceConnectionRoute guest_ip: NonEmptyStr guest_user: NonEmptyStr host_key: NonEmptyStr @@ -67,29 +64,26 @@ fn workspace_ed25519_key(line: String, offset: Int) -> String? { // This pure join consumes an invocation-bound console and an actual SSH key exchange. // A live TCP port or an address inherited from a previous incarnation cannot satisfy it. fn workspace_connection_from_observations( - identity: AttemptIdentity, gateway: WorkspaceGateway, systemd_invocation: NonEmptyStr, - console: String, scanned_keys: String, gateway_known_host: NonEmptyStr, + identity: AttemptIdentity, route: WorkspaceConnectionRoute, systemd_invocation: NonEmptyStr, + console: String, scanned_keys: String, ) -> WorkspaceConnectionReading { - if !starts_with(s: gateway_known_host as String, prefix: concat(gateway.tailnet_name as String, " ")) - || string_contains(s: gateway_known_host as String, pattern: "\n") || string_contains(s: gateway_known_host as String, pattern: "\r") { - WorkspaceConnectionUnavailable { reason: "enrolled gateway key does not name this selected host" } - } else if !runner_slot_identity_equal(a: identity.attempt.slot, b: gateway.slot) { - WorkspaceConnectionUnavailable { reason: "gateway does not name this allocation's selected host and slot" } - } else { match identity.workload { + match workspace_connection_route_refusal(slot: identity.attempt.slot, route: route) { + Present { value: reason } => WorkspaceConnectionUnavailable { reason: reason as String } + Absent => match identity.workload { WorkspaceWorkload { bootstrap_identity: bootstrap, image_identity: _, lease_expires_unix: _ } => match workspace_console_value(console: console, marker: "gunbc-workspace-identity=") { Absent => WorkspaceConnectionUnavailable { reason: "no unique invocation-bound workspace bootstrap identity" } Present { value: observed } => if observed != (bootstrap as String) { WorkspaceConnectionUnavailable { reason: "guest bootstrap identity belongs to another allocation" } - } else { workspace_connection_after_boot(identity: identity, gateway: gateway, systemd_invocation: systemd_invocation, console: console, scanned_keys: scanned_keys, gateway_known_host: gateway_known_host) } + } else { workspace_connection_after_boot(identity: identity, route: route, systemd_invocation: systemd_invocation, console: console, scanned_keys: scanned_keys) } } GitHubRunnerWorkload { registration_id: _ } => WorkspaceConnectionUnavailable { reason: "attempt is not a workspace workload" } } } } fn workspace_connection_after_boot( - identity: AttemptIdentity, gateway: WorkspaceGateway, systemd_invocation: NonEmptyStr, - console: String, scanned_keys: String, gateway_known_host: NonEmptyStr, + identity: AttemptIdentity, route: WorkspaceConnectionRoute, systemd_invocation: NonEmptyStr, + console: String, scanned_keys: String, ) -> WorkspaceConnectionReading { match workspace_console_value(console: console, marker: "gunbc-workspace-host-key=") { Absent => WorkspaceConnectionUnavailable { reason: "workspace host key has not been observed for this invocation" } @@ -108,7 +102,7 @@ fn workspace_connection_after_boot( } else { WorkspaceConnectionReady { connection: WorkspaceConnection { identity: identity, systemd_invocation: systemd_invocation, - host_account: gateway.host_account, host_tailnet: gateway.tailnet_name, gateway_known_host: gateway_known_host, + route: route, guest_ip: guest_ip as NonEmptyStr, guest_user: "root", host_key: key as NonEmptyStr, host_key_alias: concat("workspace-", serialize_content_hash(hash: content_hash_of_value(value: serialize_json(v: encode_attempt(a: in_flight_attempt_of_identity(identity: identity))) as NonEmptyStr)) as String) as NonEmptyStr, } } @@ -119,7 +113,7 @@ fn workspace_connection_after_boot( } // Called while the shared controller is running. No frontend request performs these reads. -fn workspace_observe_connection(identity: AttemptIdentity, launch: AttemptLaunch, gateway: WorkspaceGateway, gateway_known_host: NonEmptyStr) -> WorkspaceConnectionReading { +fn workspace_observe_connection(identity: AttemptIdentity, launch: AttemptLaunch, route: WorkspaceConnectionRoute) -> WorkspaceConnectionReading { match bind_attempt_console(identity: identity, launch: launch) { AttemptConsoleBound { unit: unit, systemd_invocation: invocation } => { let log = systemd.Journalctl.UnitInvocationLog(unit: unit, invocation_id: invocation) @@ -132,7 +126,7 @@ fn workspace_observe_connection(identity: AttemptIdentity, launch: AttemptLaunch AttemptConsoleBound { unit: unit_after, systemd_invocation: after } => if unit_after != unit || after != invocation || !systemd.Systemctl.IsActive(unit: unit).active { WorkspaceConnectionUnavailable { reason: "workspace incarnation stopped or changed during SSH observation" } } else { - workspace_connection_from_observations(identity: identity, gateway: gateway, systemd_invocation: invocation, console: log.stdout, scanned_keys: scanned, gateway_known_host: gateway_known_host) + workspace_connection_from_observations(identity: identity, route: route, systemd_invocation: invocation, console: log.stdout, scanned_keys: scanned) } AttemptConsoleUnbound { reason: _ } => WorkspaceConnectionUnavailable { reason: "workspace incarnation is no longer observable" } AttemptConsoleNotStarted => WorkspaceConnectionUnavailable { reason: "workspace incarnation is no longer observable" } @@ -150,12 +144,23 @@ fn workspace_observe_connection(identity: AttemptIdentity, launch: AttemptLaunch fn workspace_connection_text(connection: WorkspaceConnection) -> String { let host_alias = connection.host_key_alias as String let known_host = join([host_alias, connection.host_key as String], " ") - let gateway = join([connection.host_account as String, "@", connection.host_tailnet as String], "") let target = join([connection.guest_user as String, "@", connection.guest_ip as String], "") + let route_words = match connection.route { + WorkspaceDirectSsh { slot: _, identity_file } => concat(" -o IdentitiesOnly=yes -o IdentityAgent=none -i ", posix_single_quote(spelling: identity_file as String)) + WorkspaceGatewaySsh { gateway, known_host: _ } => concat(" -J ", posix_single_quote(spelling: join([gateway.host_account as String, "@", gateway.tailnet_name as String], ""))) + } + let gateway_key = match connection.route { + WorkspaceDirectSsh { slot: _, identity_file: _ } => "" + WorkspaceGatewaySsh { gateway: _, known_host: key } => concat(posix_single_quote(spelling: key as String), " ") + } + let location = match connection.route { + WorkspaceDirectSsh { slot, identity_file: _ } => concat("Run on ", concat(slot.host as String, ":\n")) + WorkspaceGatewaySsh { gateway: _, known_host: _ } => "" + } join([ - "mkdir -p ~/.ssh && chmod 700 ~/.ssh && printf '%s\\n' ", posix_single_quote(spelling: connection.gateway_known_host as String), " ", posix_single_quote(spelling: known_host), - " >> ~/.ssh/known_hosts && ssh -o StrictHostKeyChecking=yes -o ", posix_single_quote(spelling: concat("HostKeyAlias=", host_alias)), - " -J ", posix_single_quote(spelling: gateway), " ", posix_single_quote(spelling: target), + location, "mkdir -p ~/.ssh && chmod 700 ~/.ssh && printf '%s\\n' ", gateway_key, posix_single_quote(spelling: known_host), + " >> ~/.ssh/known_hosts && ssh -o ForwardAgent=no -o StrictHostKeyChecking=yes -o ", posix_single_quote(spelling: concat("HostKeyAlias=", host_alias)), + route_words, " ", posix_single_quote(spelling: target), "\n\nVerify allocation: cat /etc/gunbc-allocation\nExpected identity: ", connection.identity.attempt.attempt_id.value as String, "\nRelease or expiry permanently deletes this workspace disk.", ], "") diff --git a/dag/gunbc/workspace_allocation_admission.dag b/dag/gunbc/workspace_allocation_admission.dag index 4494592c293..54162670c1e 100644 --- a/dag/gunbc/workspace_allocation_admission.dag +++ b/dag/gunbc/workspace_allocation_admission.dag @@ -9,6 +9,7 @@ import extdeps.http.server { RequestSecurityContext } import gunbc.principal_projection { PrincipalRef, principal_ref_eq, principal_ref_label } import std.list { distinct_by_key } import extdeps.ssh { SshPublicKey, Ed25519, ssh_ed25519_key_identity } +import gunbc.workspace_ssh_access { WorkspaceSshAccess, WorkspaceSshFromHost, WorkspaceSshThroughGateway } import gunbc.roadmap_allocation_location { allocation_page_path } import gunbc.auth.request_authentication { RequestAccessPath, RequestAuthenticationEvidence, GoogleOidcSession, FabricServiceEvidence, SessionStandingInvalid, AuthAbsent, DeploymentAccessPolicy, @@ -21,7 +22,7 @@ import gunbc.auth.request_authentication { RequestAccessPath, // it. The roster and catalog are administrator-authored and unadmitted; only workspace_policy_admit // mints the policy every consumer reads, so a request names no key, path or OpenSSH option and a // shared, duplicated or dangling enrollment cannot reach any consumer. -type WorkspaceOwnerEnrollment { principal: PrincipalRef, public_key_ref: NonEmptyStr } +type WorkspaceOwnerEnrollment { principal: PrincipalRef, public_key_ref: NonEmptyStr, ssh_access: WorkspaceSshAccess } type WorkspaceEnrollmentRoster { enrollments: List } type WorkspacePublicKeyCatalogRow { reference: NonEmptyStr, key: SshPublicKey } type WorkspaceAuthorizationPolicy sole_constructor { enrollments: List, catalog: List } @@ -54,6 +55,12 @@ fn workspace_policy_admit(roster: WorkspaceEnrollmentRoster, catalog: List r.key.material as String)) != count(catalog) { WorkspacePolicyRefused { reason: "two workspace key references carry the same key material" } + } else if any(roster.enrollments, e => match e.ssh_access { + WorkspaceSshThroughGateway => false + WorkspaceSshFromHost { identity_file } => !starts_with(s: identity_file as String, prefix: "/") + || string_contains(s: identity_file as String, pattern: "\n") || string_contains(s: identity_file as String, pattern: "\r") + }) { + WorkspacePolicyRefused { reason: "host-local SSH requires an absolute single-line identity-file path" } } else { WorkspacePolicyAdmitted { policy: WorkspaceAuthorizationPolicy { enrollments: roster.enrollments, catalog: catalog } } } diff --git a/dag/gunbc/workspace_allocation_policy.dag b/dag/gunbc/workspace_allocation_policy.dag index 17148695f92..55bbf46a9b7 100644 --- a/dag/gunbc/workspace_allocation_policy.dag +++ b/dag/gunbc/workspace_allocation_policy.dag @@ -1,9 +1,11 @@ module gunbc.workspace_allocation_policy -import std.types { NonEmptyStr, List } -import gunbc.fleet_ssh_access { operator_ssh_principal } +import std.types { String, NonEmptyStr, List } +import extdeps.ssh { SshPublicKey, Ed25519 } +import gunbc.fleet_posix_accounts { fleet_posix_home_directory, fleet_posix_operator_user } +import gunbc.workspace_ssh_access { WorkspaceSshAccess, WorkspaceSshFromHost } import gunbc.principal_projection { PrincipalRef } -import gunbc.workspace_allocation_admission { WorkspaceEnrollmentRoster, WorkspaceOwnerEnrollment, WorkspacePublicKeyCatalogRow, WorkspacePolicyAdmission, workspace_policy_admit } +import gunbc.workspace_allocation_admission { WorkspaceEnrollmentRoster, WorkspaceOwnerEnrollment, WorkspacePublicKeyCatalogRow, WorkspacePolicyAdmission, workspace_policy_admit, workspace_owner_enrollment, WorkspacePolicyAdmitted, WorkspacePolicyRefused } // Enrollment is administrator-authored and lands as a reviewed row; the installed release is the // root-owned authority the controller reads, so no request, browser or guest can add a row. @@ -17,22 +19,35 @@ fn workspace_operator_principal() -> PrincipalRef { } } -data workspace_operator_public_key_ref: NonEmptyStr = "gunbc.fleet_ssh_access.operator_ssh_principal" +data workspace_operator_public_key_ref: NonEmptyStr = "gunbc.workspace_allocation_policy.workspace_operator_guest_key" + +// Generated on srv1 for this workspace only, at the operator's request. The private key remains +// in the operator's 0600 file there; it is not installed into host authorized_keys or workflow jobs. +data workspace_operator_guest_key: SshPublicKey = SshPublicKey { + algorithm: Ed25519, + material: "AAAAC3NzaC1lZDI1NTE5AAAAIMqh2/pEx3PptwZqnFneipNltQcry0TBF2Emi7g/BjUY", + comment: "gunbc-workspace-srv1-13", +} + +fn workspace_operator_guest_identity_file() -> NonEmptyStr { + concat(fleet_posix_home_directory(login: fleet_posix_operator_user.name) as String, + "/.ssh/gunbc-workspace-srv1-13") as NonEmptyStr +} fn workspace_enrollment_roster() -> WorkspaceEnrollmentRoster { WorkspaceEnrollmentRoster { enrollments: [ - WorkspaceOwnerEnrollment { principal: workspace_operator_principal(), public_key_ref: workspace_operator_public_key_ref }, + WorkspaceOwnerEnrollment { principal: workspace_operator_principal(), public_key_ref: workspace_operator_public_key_ref, + ssh_access: WorkspaceSshFromHost { identity_file: workspace_operator_guest_identity_file() } }, ] } } -// Each reference names exactly one enrolled ed25519 key, carried in the extdeps.ssh key model. The operator's guest key reuses the -// existing fleet SSH principal's public key; a customer's row carries that customer's own key, -// and a reference is never shared between owners. +// Each reference names exactly one enrolled ed25519 key, carried in the extdeps.ssh key model. +// Guest enrollment is independent of the credentials used to log into a fleet host. fn workspace_public_key_catalog() -> List { [ WorkspacePublicKeyCatalogRow { reference: workspace_operator_public_key_ref, - key: operator_ssh_principal.public_key, + key: workspace_operator_guest_key, }, ] } @@ -42,3 +57,13 @@ fn workspace_public_key_catalog() -> List { fn workspace_installed_policy() -> WorkspacePolicyAdmission { workspace_policy_admit(roster: workspace_enrollment_roster(), catalog: workspace_public_key_catalog()) } + +fn workspace_enrolled_ssh_access(owner: PrincipalRef) -> WorkspaceSshAccess? { + match workspace_installed_policy() { + WorkspacePolicyRefused { reason: _ } => none + WorkspacePolicyAdmitted { policy } => match workspace_owner_enrollment(policy: policy, owner: owner) { + Absent => none + Present { value: enrollment } => Present { value: enrollment.ssh_access } + } + } +} diff --git a/dag/gunbc/workspace_gateway_operator.dag b/dag/gunbc/workspace_gateway_operator.dag deleted file mode 100644 index 634ab51e260..00000000000 --- a/dag/gunbc/workspace_gateway_operator.dag +++ /dev/null @@ -1,76 +0,0 @@ -module gunbc.workspace_gateway_operator - -import std.types { String, NonEmptyStr, Int, Timestamp } -import std.process { ProcessExit, ExitSuccess, exit_failure } -import std.measure { second } -import std.content_hash { content_hash_of_value, serialize_content_hash } -import extdeps.shell -import extdeps.filesystem.filesystem_io { Filesystem } -import extdeps.tools.sleep { sleep_delay_seconds_second_carrier_projection } -import extdeps.languages.json.emit { serialize_json } -import gunbc.clock_read { clock_now_probed_at } -import gunbc.auth.approval_capability { utc_instant_seconds_after } -import gunbc.fabric_storage_binding { fabric_state_storage_binding } -import gunbc.fleet_known_hosts_anchor { FleetOpenSshTrustPolicy, fleet_ssh_attempt_identity, materialize_fleet_known_hosts_anchor, fleet_openssh_trust_policy_of } -import gunbc.workspace_operator_request { workspace_operator_on_candidate_host, workspace_operator_is_enrolled } -import gunbc.runner_slot_allocation { workspace_execution_slot_identities } -import gunbc.workspace_gateway_observer { workspace_refresh_gateway_wet } -import gunbc.workspace_gateway_store { WorkspaceGatewayEvidenceRead, WorkspaceGatewayEvidenceUnavailable } -import gunbc.workspace_gateway_admission { workspace_gateway_refusal, workspace_gateway_evidence_json } - -// A foreground operator command, bounded to one hour. It maintains the existing 60-second -// forwarding observation while an operator exercises a workspace. It owns no lease or VM -// effects, installs no service or timer, and the root controller never receives the agent. -fn workspace_gateway_refresh_operator_wet(seconds: String) -> ProcessExit { - match parse_int(s: seconds) { - Absent => exit_failure(reason: "GatewayRefreshRefused: canonical duration in seconds required") - Present { value: duration } => if duration < 1 || duration > 3600 || to_string(duration) != seconds { - exit_failure(reason: "GatewayRefreshRefused: duration must be 1..3600 seconds") - } else if !workspace_operator_on_candidate_host() || !workspace_operator_is_enrolled() { - exit_failure(reason: "GatewayRefreshRefused: operator enrollment and workspace host required") - } else { match shell.Env.Get(name: "SSH_AUTH_SOCK").value { - Absent => exit_failure(reason: "GatewayRefreshRefused: select the operator SSH agent") - Present { value: agent } => if agent == "" { exit_failure(reason: "GatewayRefreshRefused: operator agent is empty") } - else { match clock_now_probed_at() { - Absent => exit_failure(reason: "GatewayRefreshRefused: clock unavailable") - Present { value: started } => match fleet_ssh_attempt_identity(raw: concat("workspace-gateway-", serialize_content_hash(hash: content_hash_of_value(value: started)))) { - Absent => exit_failure(reason: "GatewayRefreshRefused: attempt identity invalid") - Present { value: attempt } => match fleet_openssh_trust_policy_of(outcome: materialize_fleet_known_hosts_anchor(attempt: attempt)) { - Absent => exit_failure(reason: "GatewayRefreshRefused: enrolled host keys could not be materialized") - Present { value: trust } => workspace_gateway_refresh_until(started: started, duration: duration, - trust: trust, agent_socket: agent as NonEmptyStr) - } - } - } } - } } - } -} - -fn workspace_gateway_refresh_until(started: Timestamp, duration: Int, trust: FleetOpenSshTrustPolicy, - agent_socket: NonEmptyStr) -> ProcessExit { - match clock_now_probed_at() { - Absent => exit_failure(reason: "GatewayRefreshRefused: clock unavailable") - Present { value: now } => { - let elapsed = utc_instant_seconds_after(later: now, earlier: started) - if elapsed < 0 { exit_failure(reason: "GatewayRefreshRefused: clock moved backwards") } - else if elapsed >= duration { ExitSuccess } - else { - let refusals = flat_map(workspace_execution_slot_identities(), slot => - match workspace_refresh_gateway_wet(store: fabric_state_storage_binding(), slot: slot, trust: trust, agent_socket: agent_socket) { - WorkspaceGatewayEvidenceUnavailable { reason } => [reason as String] - WorkspaceGatewayEvidenceRead { evidence } => match workspace_gateway_refusal(slot: slot, evidence: evidence, now: evidence.observed_at) { - Present { value: reason } => [reason as String] - Absent => { - let receipt = Filesystem.Write(path: "target/workspace-gateway.json", content: concat(serialize_json(v: workspace_gateway_evidence_json(evidence: evidence)), "\n")) - if receipt.success { [] } else { [receipt.error] } - } - } - }) - if count(refusals) != 0 { exit_failure(reason: join(refusals, "; ")) } - else if !sleep_delay_seconds_second_carrier_projection(duration: second(count: 15)) { - exit_failure(reason: "GatewayRefreshRefused: wait failed") - } else { workspace_gateway_refresh_until(started: started, duration: duration, trust: trust, agent_socket: agent_socket) } - } - } - } -} diff --git a/dag/gunbc/workspace_host_offer_observe.dag b/dag/gunbc/workspace_host_offer_observe.dag index 96467524f43..68645ffe87e 100644 --- a/dag/gunbc/workspace_host_offer_observe.dag +++ b/dag/gunbc/workspace_host_offer_observe.dag @@ -23,6 +23,8 @@ import gunbc.workspace_tool_observer { workspace_observe_tools, WorkspaceToolsRe import gunbc.workspace_allocation_supply { workspace_observe_host_supply, WorkspaceHostSupplyObserved, WorkspaceHostSupplyUnavailable } import gunbc.workspace_gateway_store { workspace_read_gateway_evidence, WorkspaceGatewayEvidenceRead, WorkspaceGatewayEvidenceUnavailable } import gunbc.workspace_offer_producer { WorkspaceHostOfferRead, WorkspaceHostOfferUnmet, workspace_host_offer_admit } +import gunbc.workspace_allocation_policy { workspace_enrolled_ssh_access } +import gunbc.workspace_ssh_access { WorkspaceSshFromHost, WorkspaceSshThroughGateway, WorkspaceHostAccessObserved, WorkspaceGatewayAccessObserved } // RUNS ON THE CANDIDATE HOST, inside its installed controller release, and only reads. Every // existing host-local observer is reused unchanged; this fold composes them per designated slot. @@ -55,12 +57,24 @@ fn workspace_observe_slot_offer(request: WorkspaceRequest, host: HostIdentity, s WorkspaceToolsUnobserved { reason } => WorkspaceHostOfferUnmet { host: host, reason: reason } WorkspaceToolsReady { observation: tools } => match workspace_observe_host_supply(host: host) { WorkspaceHostSupplyUnavailable { reason } => WorkspaceHostOfferUnmet { host: host, reason: reason } - WorkspaceHostSupplyObserved { observation: supply } => match workspace_read_gateway_evidence(store: fabric_state_storage_binding(), slot: slot) { - WorkspaceGatewayEvidenceUnavailable { reason } => WorkspaceHostOfferUnmet { host: host, reason: reason } - WorkspaceGatewayEvidenceRead { evidence: gateway } => match clock_now_probed_at() { - Absent => WorkspaceHostOfferUnmet { host: host, reason: "host offer clock is unavailable" } - Present { value: now } => workspace_host_offer_admit(root: fabric_control_plane_store_root(), request: request, - commissioned: committed, image: image, host: supply, tools: tools, gateway: gateway, now: now as String) + WorkspaceHostSupplyObserved { observation: supply } => match workspace_enrolled_ssh_access(owner: request.owner) { + Absent => WorkspaceHostOfferUnmet { host: host, reason: "workspace owner has no admitted SSH access policy" } + Present { value: access } => match access { + WorkspaceSshFromHost { identity_file: _ } => match clock_now_probed_at() { + Absent => WorkspaceHostOfferUnmet { host: host, reason: "host offer clock is unavailable" } + Present { value: now } => workspace_host_offer_admit(root: fabric_control_plane_store_root(), request: request, + commissioned: committed, image: image, host: supply, tools: tools, + access: WorkspaceHostAccessObserved { host: host, observed_at: supply.observed_at }, now: now as String) + } + WorkspaceSshThroughGateway => match workspace_read_gateway_evidence(store: fabric_state_storage_binding(), slot: slot) { + WorkspaceGatewayEvidenceUnavailable { reason } => WorkspaceHostOfferUnmet { host: host, reason: reason } + WorkspaceGatewayEvidenceRead { evidence: gateway } => match clock_now_probed_at() { + Absent => WorkspaceHostOfferUnmet { host: host, reason: "host offer clock is unavailable" } + Present { value: now } => workspace_host_offer_admit(root: fabric_control_plane_store_root(), request: request, + commissioned: committed, image: image, host: supply, tools: tools, + access: WorkspaceGatewayAccessObserved { evidence: gateway }, now: now as String) + } + } } } } diff --git a/dag/gunbc/workspace_offer_producer.dag b/dag/gunbc/workspace_offer_producer.dag index 4cdfcd83da1..b3b8b8f32fd 100644 --- a/dag/gunbc/workspace_offer_producer.dag +++ b/dag/gunbc/workspace_offer_producer.dag @@ -16,7 +16,8 @@ import gunbc.runner_microvm { gunbc_runner_microvm_declared_realization_allowanc import gunbc.workspace_tool_observer { WorkspaceToolsObserved, workspace_tools_refusal } import gunbc.workspace_allocation { WorkspaceRequest, workspace_execution_requirements, workspace_profile_vcpus, workspace_profile_memory } import gunbc.workspace_allocation_supply { WorkspaceHostSupplyObservation, workspace_slot_observation_refusal } -import gunbc.workspace_gateway_admission { WorkspaceGatewayEvidence, workspace_gateway_refusal } +import gunbc.workspace_ssh_access { WorkspaceAccessEvidence, workspace_access_refusal, workspace_access_receipt } +import gunbc.workspace_allocation_policy { workspace_enrolled_ssh_access } import gunbc.fabric_control_plane { fabric_cell_allocation_key } import product.placement_supply { HostIdentity } import std.algebra { trim } @@ -48,7 +49,7 @@ type WorkspaceHostOfferAdmission sole_constructor { slot: RunnerSlotIdentity observed_at: Timestamp commissioning_plan_hash: NonEmptyStr - gateway_receipt: NonEmptyStr + access_receipt: NonEmptyStr image: NonEmptyStr request_key: NonEmptyStr } @@ -59,26 +60,31 @@ type WorkspaceHostOfferRead fn workspace_host_offer_admit(root: NonEmptyStr, request: WorkspaceRequest, commissioned: WorkspaceCommittedCommissioning, image: VerifiedWorkspaceImage, host: WorkspaceHostSupplyObservation, tools: WorkspaceToolsObserved, - gateway: WorkspaceGatewayEvidence, now: Timestamp) -> WorkspaceHostOfferRead { - let guest_bytes = byte_size_count(b: workspace_profile_memory(profile: request.profile)) - let realization_bytes = byte_size_count(b: gibibyte_to_byte_size(g: gunbc_runner_microvm_declared_realization_allowance)) - if image.identity != request.image { WorkspaceHostOfferUnmet { host: host.host, reason: "installed image does not match admitted request" } } - else if hardware_thread_count_value(t: commissioned.admitted.cpu_budget) < workspace_profile_vcpus(profile: request.profile) - || byte_size_count(b: commissioned.admitted.attempt_envelope) < guest_bytes + realization_bytes { - WorkspaceHostOfferUnmet { host: host.host, reason: "commissioned exclusive slot budget cannot contain the selected guest and realization reserve" } - } else if (match workspace_tools_refusal(observation: tools, slot: commissioned.admitted.slot, now: now) { Present { value: _ } => true Absent => false }) { - WorkspaceHostOfferUnmet { host: host.host, reason: "controller tools are stale, unobserved, or belong to another host" } - } else { match workspace_slot_observation_refusal(root: root, slot: commissioned.admitted.slot, observation: host, now: now) { - Present { value: detail } => WorkspaceHostOfferUnmet { host: host.host, reason: detail } - Absent => match workspace_gateway_refusal(slot: commissioned.admitted.slot, evidence: gateway, now: now) { - Present { value: detail } => WorkspaceHostOfferUnmet { host: host.host, reason: detail } - Absent => WorkspaceHostOfferAdmitted { admission: WorkspaceHostOfferAdmission { - host: host.host, slot: commissioned.admitted.slot, observed_at: host.observed_at, - commissioning_plan_hash: commissioned.admitted.commissioning_plan_hash as NonEmptyStr, - gateway_receipt: gateway.receipt, image: image.identity, request_key: request.idempotency_key, + access: WorkspaceAccessEvidence, now: Timestamp) -> WorkspaceHostOfferRead { + match workspace_enrolled_ssh_access(owner: request.owner) { + Absent => WorkspaceHostOfferUnmet { host: host.host, reason: "workspace owner has no admitted SSH access policy" } + Present { value: required_access } => { + let guest_bytes = byte_size_count(b: workspace_profile_memory(profile: request.profile)) + let realization_bytes = byte_size_count(b: gibibyte_to_byte_size(g: gunbc_runner_microvm_declared_realization_allowance)) + if image.identity != request.image { WorkspaceHostOfferUnmet { host: host.host, reason: "installed image does not match admitted request" } } + else if hardware_thread_count_value(t: commissioned.admitted.cpu_budget) < workspace_profile_vcpus(profile: request.profile) + || byte_size_count(b: commissioned.admitted.attempt_envelope) < guest_bytes + realization_bytes { + WorkspaceHostOfferUnmet { host: host.host, reason: "commissioned exclusive slot budget cannot contain the selected guest and realization reserve" } + } else if (match workspace_tools_refusal(observation: tools, slot: commissioned.admitted.slot, now: now) { Present { value: _ } => true Absent => false }) { + WorkspaceHostOfferUnmet { host: host.host, reason: "controller tools are stale, unobserved, or belong to another host" } + } else { match workspace_slot_observation_refusal(root: root, slot: commissioned.admitted.slot, observation: host, now: now) { + Present { value: detail } => WorkspaceHostOfferUnmet { host: host.host, reason: detail } + Absent => match workspace_access_refusal(access: required_access, slot: commissioned.admitted.slot, evidence: access, now: now) { + Present { value: detail } => WorkspaceHostOfferUnmet { host: host.host, reason: detail } + Absent => WorkspaceHostOfferAdmitted { admission: WorkspaceHostOfferAdmission { + host: host.host, slot: commissioned.admitted.slot, observed_at: host.observed_at, + commissioning_plan_hash: commissioned.admitted.commissioning_plan_hash as NonEmptyStr, + access_receipt: workspace_access_receipt(evidence: access), image: image.identity, request_key: request.idempotency_key, + } } + } } } } - } } + } } fn workspace_host_offer_json(read: WorkspaceHostOfferRead) -> JsonValue { @@ -89,7 +95,7 @@ fn workspace_host_offer_json(read: WorkspaceHostOfferRead) -> JsonValue { json_kv(key: "slot_index", value: json_int(n: a.slot.slot_index)), json_kv(key: "observed_at", value: json_string(s: a.observed_at)), json_kv(key: "commissioning_plan_hash", value: json_string(s: a.commissioning_plan_hash as String)), - json_kv(key: "gateway_receipt", value: json_string(s: a.gateway_receipt as String)), + json_kv(key: "access_receipt", value: json_string(s: a.access_receipt as String)), json_kv(key: "image", value: json_string(s: a.image as String)), json_kv(key: "request_key", value: json_string(s: a.request_key as String)), ]) @@ -101,7 +107,7 @@ fn workspace_host_offer_json(read: WorkspaceHostOfferRead) -> JsonValue { } } -data workspace_host_offer_schema: String = "workspace-host-offer/v1" +data workspace_host_offer_schema: String = "workspace-host-offer/v2" fn workspace_host_offer_text(read: WorkspaceHostOfferRead) -> String { serialize_json(v: workspace_host_offer_json(read: read)) } @@ -132,14 +138,14 @@ fn workspace_host_offer_decode(text: String, expected_host: HostIdentity) -> Wor let slot = RunnerSlotIdentity { host: expected_host, slot_index: index } if !identity_is_workspace_execution_member(identity: slot) { WorkspaceHostOfferUndecodable { host: expected_host, reason: "host offer names a slot outside the workspace roster" } } else if workspace_host_offer_field(doc: doc, field: "observed_at") == "" || workspace_host_offer_field(doc: doc, field: "commissioning_plan_hash") == "" - || workspace_host_offer_field(doc: doc, field: "gateway_receipt") == "" || workspace_host_offer_field(doc: doc, field: "image") == "" + || workspace_host_offer_field(doc: doc, field: "access_receipt") == "" || workspace_host_offer_field(doc: doc, field: "image") == "" || workspace_host_offer_field(doc: doc, field: "request_key") == "" { WorkspaceHostOfferUndecodable { host: expected_host, reason: "host offer is missing a required field" } } else { let candidate = WorkspaceHostOfferAdmitted { admission: WorkspaceHostOfferAdmission { host: expected_host, slot: slot, observed_at: workspace_host_offer_field(doc: doc, field: "observed_at"), commissioning_plan_hash: workspace_host_offer_field(doc: doc, field: "commissioning_plan_hash") as NonEmptyStr, - gateway_receipt: workspace_host_offer_field(doc: doc, field: "gateway_receipt") as NonEmptyStr, + access_receipt: workspace_host_offer_field(doc: doc, field: "access_receipt") as NonEmptyStr, image: workspace_host_offer_field(doc: doc, field: "image") as NonEmptyStr, request_key: workspace_host_offer_field(doc: doc, field: "request_key") as NonEmptyStr, } } @@ -185,5 +191,5 @@ fn workspace_observed_offer_identity(admission: WorkspaceHostOfferAdmission, req supplier: NonEmptyStr) -> FabricIdentity { FabricIdentity { principal: supplier, key: serialize_content_hash(hash: content_hash_of_value(value: concat( admission.commissioning_plan_hash as String, fabric_cell_allocation_key(slot: admission.slot) as String, - request.image as String, to_string(request.profile), admission.gateway_receipt as String, admission.observed_at))) as OfferKey } + request.image as String, to_string(request.profile), admission.access_receipt as String, admission.observed_at))) as OfferKey } } diff --git a/dag/gunbc/workspace_ssh_access.dag b/dag/gunbc/workspace_ssh_access.dag new file mode 100644 index 00000000000..b0ec47225f7 --- /dev/null +++ b/dag/gunbc/workspace_ssh_access.dag @@ -0,0 +1,65 @@ +module gunbc.workspace_ssh_access + +import std.types { String, NonEmptyStr, Timestamp } +import std.content_hash { content_hash_of_value, serialize_content_hash } +import product.host_identity { HostIdentity } +import gunbc.build_cache_instance { RunnerSlotIdentity, runner_slot_identity_equal } +import gunbc.workspace_gateway { WorkspaceGateway } +import gunbc.workspace_gateway_admission { WorkspaceGatewayEvidence, workspace_gateway_refusal } +import gunbc.auth.approval_capability { utc_instant_is_canonical, utc_instant_seconds_after } + +// Administrator-owned enrollment chooses where the SSH client runs. A host-local client needs +// no forwarding permission or agent on a relay. Its identity file is local to that selected host; +// only the matching PUBLIC key is carried by the guest enrollment. +type WorkspaceSshAccess + = WorkspaceSshFromHost { identity_file: NonEmptyStr } + | WorkspaceSshThroughGateway + +type WorkspaceAccessEvidence + = WorkspaceHostAccessObserved { host: HostIdentity, observed_at: Timestamp } + | WorkspaceGatewayAccessObserved { evidence: WorkspaceGatewayEvidence } + +fn workspace_access_refusal(access: WorkspaceSshAccess, slot: RunnerSlotIdentity, + evidence: WorkspaceAccessEvidence, now: Timestamp) -> NonEmptyStr? { + match access { + WorkspaceSshFromHost { identity_file: _ } => match evidence { + WorkspaceGatewayAccessObserved { evidence: _ } => Present { value: "host-local access requires an observation on the selected host" } + WorkspaceHostAccessObserved { host, observed_at } => + if host != slot.host { Present { value: "host-local access was observed on another host" } } + else if !utc_instant_is_canonical(t: now) || !utc_instant_is_canonical(t: observed_at) + || utc_instant_seconds_after(later: now, earlier: observed_at) < 0 + || utc_instant_seconds_after(later: now, earlier: observed_at) > 60 { + Present { value: "host-local access observation is stale or in the future" } + } else { none } + } + WorkspaceSshThroughGateway => match evidence { + WorkspaceHostAccessObserved { host: _, observed_at: _ } => Present { value: "host-local access cannot prove SSH forwarding" } + WorkspaceGatewayAccessObserved { evidence: gateway } => workspace_gateway_refusal(slot: slot, evidence: gateway, now: now) + } + } +} + +fn workspace_access_receipt(evidence: WorkspaceAccessEvidence) -> NonEmptyStr { + serialize_content_hash(hash: content_hash_of_value(value: to_string(evidence) as NonEmptyStr)) as NonEmptyStr +} + +// The connection carries the route it actually verified, so a host-local observation cannot +// accidentally be rendered as a ProxyJump instruction. +type WorkspaceConnectionRoute + = WorkspaceDirectSsh { slot: RunnerSlotIdentity, identity_file: NonEmptyStr } + | WorkspaceGatewaySsh { gateway: WorkspaceGateway, known_host: NonEmptyStr } + +fn workspace_connection_route_refusal(slot: RunnerSlotIdentity, route: WorkspaceConnectionRoute) -> NonEmptyStr? { + match route { + WorkspaceDirectSsh { slot: selected, identity_file: _ } => + if runner_slot_identity_equal(a: slot, b: selected) { none } + else { Present { value: "direct SSH route names another host or slot" } } + WorkspaceGatewaySsh { gateway, known_host } => + if !runner_slot_identity_equal(a: slot, b: gateway.slot) { + Present { value: "gateway does not name this allocation's selected host and slot" } + } else if !starts_with(s: known_host as String, prefix: concat(gateway.tailnet_name as String, " ")) + || string_contains(s: known_host as String, pattern: "\n") || string_contains(s: known_host as String, pattern: "\r") { + Present { value: "enrolled gateway key does not name this selected host" } + } else { none } + } +} diff --git a/dag/test/claim/runner/workspace_microvm_witness_test.dag b/dag/test/claim/runner/workspace_microvm_witness_test.dag index 476ca44bc69..067cae7a63c 100644 --- a/dag/test/claim/runner/workspace_microvm_witness_test.dag +++ b/dag/test/claim/runner/workspace_microvm_witness_test.dag @@ -8,7 +8,8 @@ import gunbc.runner_microvm_lifecycle { AttemptIdentity, in_flight_attempt_of_id import gunbc.runner_microvm_cell_readiness { WorkspaceWorkload, in_flight_attempt_same } import gunbc.runner_microvm_cell_readiness_store { encode_attempt, decode_attempt, DecodeOk } import gunbc.workspace_gateway { WorkspaceGateway } -import gunbc.workspace_microvm_access { WorkspaceConnectionReady, WorkspaceConnectionUnavailable, workspace_connection_from_observations } +import gunbc.workspace_ssh_access { WorkspaceDirectSsh, WorkspaceGatewaySsh } +import gunbc.workspace_microvm_access { WorkspaceConnectionReady, WorkspaceConnectionUnavailable, workspace_connection_from_observations, workspace_connection_text } import gunbc.runner_microvm_network { runner_slot_network } import extdeps.network.ipv4 { render_ipv4_address } import gunbc.workspace_microvm { workspace_public_key, WorkspacePublicKeyRefused, workspace_machine_config } @@ -44,7 +45,7 @@ fn test_scan(identity: AttemptIdentity, key: String) -> String { test fn workspace_readiness_requires_bootstrap_and_live_ssh_key() -> Bool { match workspace_test_identity() { Absent => false - Present { value: id } => match workspace_connection_from_observations(identity: id, gateway: test_gateway(identity: id), systemd_invocation: "systemd-test", gateway_known_host: "srv1.test.ts.net ssh-ed25519 AAAAGATEWAY", console: test_console(bootstrap: "workspace-test"), scanned_keys: test_scan(identity: id, key: "AAAATEST")) { + Present { value: id } => match workspace_connection_from_observations(identity: id, route: WorkspaceGatewaySsh { gateway: test_gateway(identity: id), known_host: "srv1.test.ts.net ssh-ed25519 AAAAGATEWAY" }, systemd_invocation: "systemd-test", console: test_console(bootstrap: "workspace-test"), scanned_keys: test_scan(identity: id, key: "AAAATEST")) { WorkspaceConnectionReady { connection: c } => c.identity.allocation_generation == 4 && (c.host_key as String) == "ssh-ed25519 AAAATEST" _ => false } @@ -54,7 +55,7 @@ test fn workspace_readiness_requires_bootstrap_and_live_ssh_key() -> Bool { test fn workspace_readiness_refuses_old_guest_at_reused_address() -> Bool { match workspace_test_identity() { Absent => false - Present { value: id } => match workspace_connection_from_observations(identity: id, gateway: test_gateway(identity: id), systemd_invocation: "systemd-test", gateway_known_host: "srv1.test.ts.net ssh-ed25519 AAAAGATEWAY", console: test_console(bootstrap: "workspace-test"), scanned_keys: test_scan(identity: id, key: "AAAAPREVIOUS")) { + Present { value: id } => match workspace_connection_from_observations(identity: id, route: WorkspaceGatewaySsh { gateway: test_gateway(identity: id), known_host: "srv1.test.ts.net ssh-ed25519 AAAAGATEWAY" }, systemd_invocation: "systemd-test", console: test_console(bootstrap: "workspace-test"), scanned_keys: test_scan(identity: id, key: "AAAAPREVIOUS")) { WorkspaceConnectionUnavailable { reason: _ } => true _ => false } @@ -64,7 +65,7 @@ test fn workspace_readiness_refuses_old_guest_at_reused_address() -> Bool { test fn workspace_readiness_refuses_foreign_bootstrap() -> Bool { match workspace_test_identity() { Absent => false - Present { value: id } => match workspace_connection_from_observations(identity: id, gateway: test_gateway(identity: id), systemd_invocation: "systemd-test", gateway_known_host: "srv1.test.ts.net ssh-ed25519 AAAAGATEWAY", console: test_console(bootstrap: "previous-allocation"), scanned_keys: test_scan(identity: id, key: "AAAATEST")) { + Present { value: id } => match workspace_connection_from_observations(identity: id, route: WorkspaceGatewaySsh { gateway: test_gateway(identity: id), known_host: "srv1.test.ts.net ssh-ed25519 AAAAGATEWAY" }, systemd_invocation: "systemd-test", console: test_console(bootstrap: "previous-allocation"), scanned_keys: test_scan(identity: id, key: "AAAATEST")) { WorkspaceConnectionUnavailable { reason: _ } => true _ => false } @@ -90,3 +91,42 @@ test fn workspace_production_machine_config_consumes_committed_grant() -> Bool { _ => false } } + + +test fn direct_ssh_uses_the_vm_key_without_a_jump_or_agent() -> Bool { + match workspace_test_identity() { + Absent => false + Present { value: id } => match workspace_connection_from_observations(identity: id, + route: WorkspaceDirectSsh { slot: id.attempt.slot, identity_file: "/fixture/vm-key" }, + systemd_invocation: "systemd-test", console: test_console(bootstrap: "workspace-test"), scanned_keys: test_scan(identity: id, key: "AAAATEST")) { + WorkspaceConnectionReady { connection } => { + let text = workspace_connection_text(connection: connection) + starts_with(s: text, prefix: "Run on srv1:") + && string_contains(s: text, pattern: "IdentityAgent=none") + && string_contains(s: text, pattern: "ForwardAgent=no") + && string_contains(s: text, pattern: "-i '/fixture/vm-key'") + && !string_contains(s: text, pattern: " -J ") + } + _ => false + } + } +} + +test fn direct_ssh_refuses_a_foreign_route_or_previous_guest_key() -> Bool { + match workspace_test_identity() { + Absent => false + Present { value: id } => { + let foreign = match workspace_connection_from_observations(identity: id, + route: WorkspaceDirectSsh { slot: RunnerSlotIdentity { host: "srv2", slot_index: 7 }, identity_file: "/fixture/vm-key" }, + systemd_invocation: "systemd-test", console: test_console(bootstrap: "workspace-test"), scanned_keys: test_scan(identity: id, key: "AAAATEST")) { + WorkspaceConnectionUnavailable { reason: _ } => true _ => false + } + let previous = match workspace_connection_from_observations(identity: id, + route: WorkspaceDirectSsh { slot: id.attempt.slot, identity_file: "/fixture/vm-key" }, + systemd_invocation: "systemd-test", console: test_console(bootstrap: "workspace-test"), scanned_keys: test_scan(identity: id, key: "AAAAPREVIOUS")) { + WorkspaceConnectionUnavailable { reason: _ } => true _ => false + } + foreign && previous + } + } +} diff --git a/dag/test/claim/runner/workspace_owner_access_witness_test.dag b/dag/test/claim/runner/workspace_owner_access_witness_test.dag index e60e6aa0035..83944090aa5 100644 --- a/dag/test/claim/runner/workspace_owner_access_witness_test.dag +++ b/dag/test/claim/runner/workspace_owner_access_witness_test.dag @@ -1,15 +1,16 @@ module test.claim.runner.workspace_owner_access_witness_test +import gunbc.workspace_ssh_access { WorkspaceSshThroughGateway, WorkspaceSshFromHost } + import std.types { Bool, NonEmptyStr, List } import std.optional { Absent, Present } import extdeps.ssh { SshPublicKey, Ed25519, Rsa, ssh_public_key_line } -import gunbc.fleet_ssh_access { operator_ssh_principal } import gunbc.principal_projection { PrincipalRef } import gunbc.workspace_allocation_admission { WorkspaceAuthorizationPolicy, WorkspaceEnrollmentRoster, WorkspaceOwnerEnrollment, WorkspacePublicKeyCatalogRow, WorkspacePolicyAdmitted, WorkspacePolicyRefused, workspace_policy_admit, } -import gunbc.workspace_allocation_policy { workspace_installed_policy, workspace_operator_principal, workspace_operator_public_key_ref } +import gunbc.workspace_allocation_policy { workspace_installed_policy, workspace_operator_principal, workspace_operator_public_key_ref, workspace_operator_guest_key } import gunbc.workspace_microvm { workspace_owner_public_key, WorkspacePublicKeyAdmitted, WorkspacePublicKeyNotAdmitted } fn alice() -> PrincipalRef { PrincipalRef { authority: "google-oidc", namespace: "accounts", subject: "alice-test" } } @@ -33,8 +34,8 @@ fn admits(roster: List, catalog: List WorkspaceAuthorizationPolicy? { match workspace_policy_admit(roster: WorkspaceEnrollmentRoster { enrollments: [ - WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }, - WorkspaceOwnerEnrollment { principal: bob(), public_key_ref: "bob-key" }, + WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }, + WorkspaceOwnerEnrollment { principal: bob(), public_key_ref: "bob-key", ssh_access: WorkspaceSshThroughGateway }, ] }, catalog: [alice_row(), bob_row()]) { WorkspacePolicyAdmitted { policy } => Present { value: policy } WorkspacePolicyRefused { reason: _ } => none @@ -44,7 +45,7 @@ fn two_owner_policy() -> WorkspaceAuthorizationPolicy? { // One enrolled owner with the shared witness reference, admitted through the production join. // Consumers of this helper match Absent as a failed claim, never as a permissive default. fn witness_single_owner_policy(owner: PrincipalRef) -> WorkspaceAuthorizationPolicy? { - match workspace_policy_admit(roster: WorkspaceEnrollmentRoster { enrollments: [WorkspaceOwnerEnrollment { principal: owner, public_key_ref: "witness-key" }] }, + match workspace_policy_admit(roster: WorkspaceEnrollmentRoster { enrollments: [WorkspaceOwnerEnrollment { principal: owner, public_key_ref: "witness-key", ssh_access: WorkspaceSshThroughGateway }] }, catalog: [key_row(reference: "witness-key", material: alice_material, comment: "witness")]) { WorkspacePolicyAdmitted { policy } => Present { value: policy } WorkspacePolicyRefused { reason: _ } => none @@ -61,32 +62,32 @@ fn witness_empty_policy() -> WorkspaceAuthorizationPolicy? { // REDs at the policy join: no list order may decide which enrollment or key line wins. test fn an_ambiguous_enrollment_roster_is_refused_at_admission() -> Bool { let shared_ref = !admits(roster: [ - WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }, - WorkspaceOwnerEnrollment { principal: bob(), public_key_ref: "alice-key" }, + WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }, + WorkspaceOwnerEnrollment { principal: bob(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }, ], catalog: [alice_row()]) let two_refs = !admits(roster: [ - WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }, - WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "bob-key" }, + WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }, + WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "bob-key", ssh_access: WorkspaceSshThroughGateway }, ], catalog: [alice_row(), bob_row()]) let duplicate_rows = !admits(roster: [ - WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }, - WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }, + WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }, + WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }, ], catalog: [alice_row()]) - let dangling_ref = !admits(roster: [WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }], catalog: [bob_row()]) - let unused_line = !admits(roster: [WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }], catalog: [alice_row(), bob_row()]) - let two_lines = !admits(roster: [WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }], + let dangling_ref = !admits(roster: [WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }], catalog: [bob_row()]) + let unused_line = !admits(roster: [WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }], catalog: [alice_row(), bob_row()]) + let two_lines = !admits(roster: [WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }], catalog: [alice_row(), key_row(reference: "alice-key", material: bob_material, comment: "other")]) let shared_bytes = !admits(roster: [ - WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }, - WorkspaceOwnerEnrollment { principal: bob(), public_key_ref: "bob-key" }, + WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }, + WorkspaceOwnerEnrollment { principal: bob(), public_key_ref: "bob-key", ssh_access: WorkspaceSshThroughGateway }, ], catalog: [alice_row(), key_row(reference: "bob-key", material: alice_material, comment: "alice")]) shared_ref && two_refs && duplicate_rows && dangling_ref && unused_line && two_lines && shared_bytes } fn alice_and_bob_with(bob_key: WorkspacePublicKeyCatalogRow) -> Bool { admits(roster: [ - WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key" }, - WorkspaceOwnerEnrollment { principal: bob(), public_key_ref: "bob-key" }, + WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", ssh_access: WorkspaceSshThroughGateway }, + WorkspaceOwnerEnrollment { principal: bob(), public_key_ref: "bob-key", ssh_access: WorkspaceSshThroughGateway }, ], catalog: [alice_row(), bob_key]) } @@ -150,15 +151,27 @@ test fn a_stale_or_foreign_key_reference_cannot_reach_another_owners_guest() -> } // Inhabitance of the installed route: the shipped roster and catalog admit, and the controller's -// resolver gives the operator exactly the fleet operator key line written into authorized_keys. +// resolver gives the operator exactly the dedicated guest key line written into authorized_keys. test fn installed_policy_resolves_the_operator_enrollment() -> Bool { match workspace_installed_policy() { WorkspacePolicyRefused { reason: _ } => false WorkspacePolicyAdmitted { policy } => match workspace_owner_public_key(policy: policy, owner: workspace_operator_principal(), reference: workspace_operator_public_key_ref) { WorkspacePublicKeyAdmitted { key: k } => (k.reference as String) == (workspace_operator_public_key_ref as String) - && (k.line as String) == (ssh_public_key_line(k: operator_ssh_principal.public_key) as String) + && (k.line as String) == (ssh_public_key_line(k: workspace_operator_guest_key) as String) WorkspacePublicKeyNotAdmitted { reason: _ } => false } } } + +fn admits_host_identity_file(path: NonEmptyStr) -> Bool { + admits(roster: [WorkspaceOwnerEnrollment { principal: alice(), public_key_ref: "alice-key", + ssh_access: WorkspaceSshFromHost { identity_file: path } }], catalog: [alice_row()]) +} + +test fn host_identity_file_requires_an_absolute_single_line_path() -> Bool { + admits_host_identity_file(path: "/fixture/vm-key") + && !admits_host_identity_file(path: "relative/vm-key") + && !admits_host_identity_file(path: "/fixture/vm-key\nsecond-line") + && !admits_host_identity_file(path: "/fixture/vm-key\rsecond-line") +} diff --git a/dag/test/claim/workspace_gateway_operator_witness_test.dag b/dag/test/claim/workspace_gateway_operator_witness_test.dag deleted file mode 100644 index 7cba8777b98..00000000000 --- a/dag/test/claim/workspace_gateway_operator_witness_test.dag +++ /dev/null @@ -1,23 +0,0 @@ -module test.claim.workspace_gateway_operator_witness_test - -import std.types { Bool, String } -import gunbc.runner_slot_allocation -import gunbc.workspace_gateway -import gunbc.fleet_known_hosts_anchor -import std.process { ExitSuccess, ExitFailure } -import gunbc.workspace_gateway_operator { workspace_gateway_refresh_operator_wet } - -test fn gateway_refresh_refuses_unbounded_or_noncanonical_durations_before_observation() -> Bool { - all(["", "0", "-1", "3601", "01", "forever"], duration => match workspace_gateway_refresh_operator_wet(seconds: duration) { - ExitFailure { code: _, reason } => starts_with(s: reason, prefix: "GatewayRefreshRefused:") - ExitSuccess => false - }) -} - -test fn workspace_gateway_tailnet_names_have_one_enrolled_host_key() -> Bool { - all(gunbc.runner_slot_allocation.workspace_execution_slot_identities(), slot => - match gunbc.workspace_gateway.workspace_fleet_gateway(slot: slot) { - Absent => false - Present { value: gateway } => count(filter(gunbc.fleet_known_hosts_anchor.fleet_known_hosts_rows(), row => row.endpoint == (gateway.tailnet_name as String))) == 1 - }) -} diff --git a/dag/test/claim/workspace_reconcile_once_witness_test.dag b/dag/test/claim/workspace_reconcile_once_witness_test.dag index 817f6da27d5..d8b56a7d5d0 100644 --- a/dag/test/claim/workspace_reconcile_once_witness_test.dag +++ b/dag/test/claim/workspace_reconcile_once_witness_test.dag @@ -25,12 +25,12 @@ fn request() -> WorkspaceRequest { fn offer_line_with(host: String, slot_index: Int, extra: Bool) -> String { let members = [ - json_kv(key: "schema", value: json_string(s: "workspace-host-offer/v1")), + json_kv(key: "schema", value: json_string(s: "workspace-host-offer/v2")), json_kv(key: "host", value: json_string(s: host)), json_kv(key: "slot_index", value: json_int(n: slot_index)), json_kv(key: "observed_at", value: json_string(s: "2026-10-04T12:00:00Z")), json_kv(key: "commissioning_plan_hash", value: json_string(s: "plan-hash")), - json_kv(key: "gateway_receipt", value: json_string(s: "gateway-receipt")), + json_kv(key: "access_receipt", value: json_string(s: "access-receipt")), json_kv(key: "image", value: json_string(s: workspace_demo_image_identity() as String)), json_kv(key: "request_key", value: json_string(s: "reconcile-test")), ] diff --git a/dag/test/claim/workspace_ssh_access_witness_test.dag b/dag/test/claim/workspace_ssh_access_witness_test.dag new file mode 100644 index 00000000000..87ad6534a08 --- /dev/null +++ b/dag/test/claim/workspace_ssh_access_witness_test.dag @@ -0,0 +1,56 @@ +module test.claim.workspace_ssh_access_witness_test + +import std.types { Bool, String } +import gunbc.build_cache_instance { RunnerSlotIdentity } +import gunbc.workspace_ssh_access { WorkspaceSshFromHost, WorkspaceSshThroughGateway, WorkspaceHostAccessObserved, workspace_access_refusal, workspace_access_receipt } +import gunbc.workspace_allocation_policy { workspace_enrolled_ssh_access, workspace_operator_principal, workspace_operator_guest_key, workspace_operator_guest_identity_file } +import gunbc.fleet_ssh_access { operator_ssh_principal, fleet_automation_ssh_key } +import gunbc.runner_slot_allocation +import gunbc.workspace_gateway +import gunbc.fleet_known_hosts_anchor + +test fn direct_host_access_is_fresh_and_bound_to_the_selected_host() -> Bool { + let slot = RunnerSlotIdentity { host: "srv1", slot_index: 13 } + let path = WorkspaceSshFromHost { identity_file: "/fixture/vm-key" } + let observed = WorkspaceHostAccessObserved { host: "srv1", observed_at: "2026-10-09T00:00:00Z" } + let positive = match workspace_access_refusal(access: path, slot: slot, evidence: observed, now: "2026-10-09T00:00:01Z") { Absent => true _ => false } + let foreign = match workspace_access_refusal(access: path, slot: RunnerSlotIdentity { host: "srv2", slot_index: 13 }, evidence: observed, now: "2026-10-09T00:00:01Z") { Present { value: _ } => true _ => false } + let stale = match workspace_access_refusal(access: path, slot: slot, evidence: observed, now: "2026-10-09T00:01:01Z") { Present { value: _ } => true _ => false } + let future = match workspace_access_refusal(access: path, slot: slot, evidence: observed, now: "2026-10-08T23:59:59Z") { Present { value: _ } => true _ => false } + positive && foreign && stale && future +} + +test fn a_host_local_observation_cannot_admit_a_gateway_route() -> Bool { + match workspace_access_refusal(access: WorkspaceSshThroughGateway, + slot: RunnerSlotIdentity { host: "srv1", slot_index: 13 }, + evidence: WorkspaceHostAccessObserved { host: "srv1", observed_at: "2026-10-09T00:00:00Z" }, now: "2026-10-09T00:00:01Z") { + Present { value: _ } => true + Absent => false + } +} + +test fn the_operator_uses_a_separate_guest_key_on_the_host() -> Bool { + let separate = workspace_operator_guest_key.material != operator_ssh_principal.public_key.material + && workspace_operator_guest_key.material != fleet_automation_ssh_key.material + separate && match workspace_enrolled_ssh_access(owner: workspace_operator_principal()) { + Present { value: WorkspaceSshFromHost { identity_file } } => identity_file == workspace_operator_guest_identity_file() + _ => false + } +} + +test fn workspace_gateway_tailnet_names_have_one_enrolled_host_key() -> Bool { + all(gunbc.runner_slot_allocation.workspace_execution_slot_identities(), slot => + match gunbc.workspace_gateway.workspace_fleet_gateway(slot: slot) { + Absent => false + Present { value: gateway } => count(filter(gunbc.fleet_known_hosts_anchor.fleet_known_hosts_rows(), row => row.endpoint == (gateway.tailnet_name as String))) == 1 + }) +} + +// The wire identity must consume the observed host and instant, not a fixed success marker. +test fn host_access_receipt_is_bound_to_its_observation() -> Bool { + let observed = WorkspaceHostAccessObserved { host: "srv1", observed_at: "2026-10-09T00:00:00Z" } + let receipt = workspace_access_receipt(evidence: observed) + receipt == workspace_access_receipt(evidence: observed) + && receipt != workspace_access_receipt(evidence: WorkspaceHostAccessObserved { host: "srv2", observed_at: "2026-10-09T00:00:00Z" }) + && receipt != workspace_access_receipt(evidence: WorkspaceHostAccessObserved { host: "srv1", observed_at: "2026-10-09T00:00:01Z" }) +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 5c2cb145684..6e2c6255e6b 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -23,7 +23,7 @@ collected that refusal; its green workflow conclusion is not a boot receipt. | Commissioning | No commissioning or readiness record | Run reviewed commissioning plan/apply and verify its protected transaction | | Protected state service | Running revision `5b96cab749088aa5cec71ce9ad52e1857724d252` predates the authenticated state route | Converge fabric storage through its existing deployment path | | State-key custody | Key exists; operator can read it; `ghrunner` cannot | Preserve custody; exact installed helpers reconcile admitted intent and read the selected allocation | -| Gateway observation | Observer has no production caller; admission expires after 60 seconds | Bounded foreground operator refresh using the enrolled agent and host keys | +| SSH access | Guest enrollment reused the personal operator key and required gateway evidence even for a client on srv1 | Enroll a dedicated guest key on srv1; model host-local SSH explicitly and retain gateway proof for gateway clients | | Allocation dispatch | No workflow mode; release path came from an unrelated environment variable | Add `workspace_allocation_plan`, bind its installed release to the admitted run revision, reuse reviewed `apply` | | Operator lifecycle | Request producer exists; release/observation need CLI entries | Add generation-checked release and owner-scoped JSON receipts under operator authority | @@ -46,10 +46,10 @@ empty commissioning prestate. Do not restore the old CI reservation service: `workspace_commissioning_plan` and its reviewed `apply` on srv1. The existing operator-local fleet plan/apply path is available for pre-merge validation; do not widen the main-only GCP login to run branch workflows. -4. Keep `workspace_gateway_refresh_operator_wet --arg seconds=3600` running in - the operator context with the enrolled key selected in `SSH_AUTH_SOCK`. It - refreshes forwarding evidence every 15 seconds and stops on refusal or at its - deadline. It does not install a scheduler. As the enrolled operator on srv1, use `workspace_operator_request_wet` with +4. The installed operator enrollment uses a dedicated guest key whose private + file stays on srv1 at `/home/briansrls/.ssh/gunbc-workspace-srv1-13`, owned by + `briansrls` with mode `0600`. No agent forwarding is required. As the enrolled + operator on srv1, use `workspace_operator_request_wet` with `GUNBC_WORKSPACE_REQUEST_KEY`, `GUNBC_WORKSPACE_REQUEST_PROFILE` and `GUNBC_WORKSPACE_REQUEST_EXPIRES_AT`. Use a canonical UTC expiry within the policy's one-hour lease. The owner and public key come from installed policy. @@ -57,8 +57,10 @@ empty commissioning prestate. Do not restore the old CI reservation service: through `workspace_operator_observe_wet` with the same key. Each operator command writes `target/workspace-operation.json`; a recorded request alone is not evidence that the guest is usable. -6. Authenticate SSH using the allocation's connection data and verified host key. - Record the guest identity and a command executed in the guest. +6. Authenticate SSH from srv1 using the allocation's connection data and verified + guest host key. The rendered command selects the dedicated identity file and + disables the agent and agent forwarding. Record the guest identity and a command + executed in the guest. 7. Run `workspace_operator_release_wet` with the key and `GUNBC_WORKSPACE_RELEASE_GENERATION` from the observation. Reconcile with another allocation plan/apply, and verify released state and absent guest. @@ -79,7 +81,14 @@ same-slot reuse with a new incarnation, expiry cleanup, and final released/ready state. Record run IDs, revisions, allocation IDs and generations here as they are observed. Never record private keys, tokens or signed state-request envelopes. -Local validation: 44 focused witnesses passed across allocation dispatch, lifecycle, +The host-local SSH revision passes all 28 focused witnesses across access policy, +owner enrollment, guest connection verification and allocation reconciliation. +The full tree (8,268 source files) parses, and formatting checks pass. An independent +`ssh -G` read on srv1 confirms the dedicated identity file, `IdentityAgent=none`, +`IdentitiesOnly=yes` and `ForwardAgent=no`. This checks client selection; it is not +a guest login receipt. + +Local validation of the preceding revision: 44 focused witnesses passed across allocation dispatch, lifecycle, launch directives, host offers, owner policy, gateway refresh, workflow dispatch and deployment candidate admission, including three deployment-readiness refusal controls and two cutover recovery/refusal controls. The generated compiler-pair build step was also executed with controlled compiler outcomes: both output streams were retained, and exit codes 0 and 37 were preserved. The image builder produced an ext4 image whose `/root` and `/usr/sbin/sshd` are owned by UID/GID 0. The fresh compiler parsed the @@ -138,15 +147,17 @@ No printer action is performed by VM bringup. The approved deployment's tree sync and independent repository readback completed at revision `19b12d88d6c0206c3f259cf85b0b664e65000b26`. Dashboard, approval broker, tailnet door and fabric storage restarted between 04:11:48 and 04:12:04 UTC; -dashboard readiness is still being checked. At 04:21 UTC the authenticated operator read returned `WorkspaceOperatorObserveRefused: request absent for operator`, proving the protected owner ledger is readable without creating an allocation. The preceding probe finished 19 seconds before the new storage listener opened and is not counted as acceptance. +dashboard readiness later passed through the existing HTTPS route. At 04:21 UTC the authenticated operator read returned `WorkspaceOperatorObserveRefused: request absent for operator`, proving the protected owner ledger is readable without creating an allocation. The preceding probe finished 19 seconds before the new storage listener opened and is not counted as acceptance. The nested deployment/readback interpreters peaked above the original 24 GiB scope bound, so that operation's cap was raised to 36 GiB with swap disabled. -The enrolled operator key is the MacBook key, fingerprint -`SHA256:KEMuhG7TCEjoaF2Q8Xn/4jJd+5uZUggzycUYnLXQojI`. It is already authorized -on srv1. The local srv1/srv2 keys differ and no agent was selected. Gateway and -guest SSH acceptance therefore require the operator's forwarded agent; its socket -has been requested. No enrollment or authorized-key policy is being widened. +The operator selected SSH from srv1. A dedicated guest-only ED25519 key was +created there, fingerprint `SHA256:sxhkzO98Tp3tSXi+nD2FJBfbRTnOYQn6QpuTlETnbQI`. +Only its public key is enrolled in source. It is not authorized for fleet-host +login or given to workflow jobs. Personal-agent forwarding was proposed earlier, +then withdrawn; no forwarded personal agent was used. Host-local access and +access through a gateway now have distinct policy and evidence types, so an +observation on srv1 cannot assert gateway forwarding permission. The deployment installed its members but returned a cutover refusal: the existing `ghrunner` deployment entry cannot read the protected-state credential. Its final @@ -161,4 +172,14 @@ installation root. Full shared-deployment convergence is not claimed. During cold startup, the shared dashboard slice exceeded its 49 GiB memory-high threshold (53 GiB hard cap) and spent about 60% of wall time stalled. Publication and belt timers/helpers were temporarily stopped within the approved interruption -window to let the services finish startup. Their restoration remains required. +window to let the services finish startup. Both timers were restored and read back active. The unused replacement +tailnet-door service is stopped after its socket-permission failure; its failed +cutover never changed the root route. An independent HTTPS `/healthz` read through +that existing route returns the approved revision and surface identity +`998c13f6282e889f`. + +Controller installation [37884238413](https://github.com/gunb-ai/gunbc/actions/runs/37884238413) +was canceled before installation at `95037360ab1040b9de4efe4671bb53cad473cb34` +when the operator selected host-local SSH. The verified image artifact from run +`37882239644` remains the image to install. No commissioning or guest allocation +has been performed yet. From 304eb9f4c5223ccc64e9c6f881294a591d89343c Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 05:36:03 +0000 Subject: [PATCH 08/19] Derive fleet workflow job selection from mode assignments --- .github/workflows/fleet-converge.yml | 2 +- dag/gunbc/fleet/fleet_converge_workflow.dag | 9 ++---- ...election_diverges_from_mode_assignment.dag | 15 ++++++++++ ...rkspace_workflow_dispatch_witness_test.dag | 20 +++++++++++-- docs/plans/srv1-workspace-vm-bringup.md | 29 ++++++++++++++++++- 5 files changed, 65 insertions(+), 10 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/workflow_job_selection_diverges_from_mode_assignment.dag diff --git a/.github/workflows/fleet-converge.yml b/.github/workflows/fleet-converge.yml index 782b58b26e7..65c371710ed 100644 --- a/.github/workflows/fleet-converge.yml +++ b/.github/workflows/fleet-converge.yml @@ -622,7 +622,7 @@ jobs: runs-on: ${{ fromJSON(format('["self-hosted","linux","arm64","{0}"]', github.event.inputs.host)) }} needs: [build] timeout-minutes: 295 - if: github.event.inputs.mode != 'mtcollins1_boot' && github.event.inputs.mode != 'gcp_iam_converge' && github.event.inputs.mode != 'namecheap_observe' && github.event.inputs.mode != 'printer' + if: github.event.inputs.mode == 'plan' || github.event.inputs.mode == 'launch_environment_plan' || github.event.inputs.mode == 'allocation_store_plan' || github.event.inputs.mode == 'workspace_commissioning_plan' || github.event.inputs.mode == 'workspace_allocation_plan' || github.event.inputs.mode == 'workspace_image_build' || github.event.inputs.mode == 'apply' || github.event.inputs.mode == 'org_actions_observe' || github.event.inputs.mode == 'app_control_plane_observe' || github.event.inputs.mode == 'microvm_host_converge' || github.event.inputs.mode == 'microvm_network_observe' || github.event.inputs.mode == 'microvm_network_apply' || github.event.inputs.mode == 'guest_image_observe' || github.event.inputs.mode == 'guest_image_converge' || github.event.inputs.mode == 'microvm_boot_probe' || github.event.inputs.mode == 'spark_grants' || github.event.inputs.mode == 'spark_bootstrap' || github.event.inputs.mode == 'spark_serving_apply' || github.event.inputs.mode == 'spark_native_serving_apply' || github.event.inputs.mode == 'spark_runtime_image_probe' || github.event.inputs.mode == 'spark_wireless_link_converge' || github.event.inputs.mode == 'spark_grants_observe' || github.event.inputs.mode == 'spark_v41_checkpoint_materialize' || github.event.inputs.mode == 'spark_v41_row_store_encode' || github.event.inputs.mode == 'spark_v41_row_store_readback' || github.event.inputs.mode == 'spark_v41_engram_differential' || github.event.inputs.mode == 'spark_runtime_image_produce' || github.event.inputs.mode == 'spark_runtime_image_distribute' || github.event.inputs.mode == 'spark_arm_group_load' || github.event.inputs.mode == 'spark_arm_checkpoint_materialize' || github.event.inputs.mode == 'spark_arm_group_observe' || github.event.inputs.mode == 'spark_arm_group_launch_plan' || github.event.inputs.mode == 'spark_arm_group_launch' || github.event.inputs.mode == 'spark_v41_serving_load' || github.event.inputs.mode == 'microvm_slot_reserve' || github.event.inputs.mode == 'microvm_slot_start' || github.event.inputs.mode == 'runner_host_file_observe' || github.event.inputs.mode == 'runner_host_file_converge' || github.event.inputs.mode == 'site_pxe_edge_observe' || github.event.inputs.mode == 'site_pxe_edge_converge' || github.event.inputs.mode == 'runner_password_session_tool_converge' || github.event.inputs.mode == 'runner_browser_toolchain_converge' || github.event.inputs.mode == 'r2_mint_preflight' || github.event.inputs.mode == 'r2_object_write_mint' || github.event.inputs.mode == 'org_runner_roster_observe' || github.event.inputs.mode == 'approval_keyring_converge' || github.event.inputs.mode == 'approval_device_enrolment_code_issue' || github.event.inputs.mode == 'mtcollins1_fan_observe' || github.event.inputs.mode == 'mtcollins1_ui_bundle_observe' || github.event.inputs.mode == 'mtcollins1_kvm_observer_observe' || github.event.inputs.mode == 'mtcollins1_census_image_publish' || github.event.inputs.mode == 'mtcollins1_census_member_readback' || github.event.inputs.mode == 'mtcollins1_census_qemu_host_observe' || github.event.inputs.mode == 'mtcollins1_census_qemu_toolchain_converge' || github.event.inputs.mode == 'host_credential_custody_converge' || github.event.inputs.mode == 'app_key_version_verify' || github.event.inputs.mode == 'r2_bucket_ensure' || github.event.inputs.mode == 'r2_bucket_admin_mint' || github.event.inputs.mode == 'r2_workspace_object_read_mint' || github.event.inputs.mode == 'r2_workspace_object_write_mint' || github.event.inputs.mode == 'r2_cache_object_read_mint' || github.event.inputs.mode == 'r2_cache_object_write_mint' || github.event.inputs.mode == 'r2_conditional_put_race_probe' || github.event.inputs.mode == 'workspace_source_pack' || github.event.inputs.mode == 'workspace_checkpoint_measure' || github.event.inputs.mode == 'workspace_checkpoint_restore' || github.event.inputs.mode == 'fabric_writer_identity_observe' || github.event.inputs.mode == 'pair_serving_d0' || github.event.inputs.mode == 'microvm_runner_group_ensure' permissions: contents: read actions: read diff --git a/dag/gunbc/fleet/fleet_converge_workflow.dag b/dag/gunbc/fleet/fleet_converge_workflow.dag index 61c219b3bb0..7d6837d971c 100644 --- a/dag/gunbc/fleet/fleet_converge_workflow.dag +++ b/dag/gunbc/fleet/fleet_converge_workflow.dag @@ -4264,12 +4264,9 @@ fn fleet_converge_consumer_prelude_steps() -> List { // repo-wide provider admits for every job in the repository. It consumes the fleet key because the // census medium readback on srv2 runs over fleet ssh; the dedicated account holds a pinned cell for // that version and for the BMC credential, and nothing else. -data fleet_converge_shared_job_if: String = join([ - "github.event.inputs.mode != '", fleet_converge_workflow_mode_wire(mode: MtCollins1Boot), "'", - " && github.event.inputs.mode != '", fleet_converge_workflow_mode_wire(mode: GcpIamConverge), "'", - " && github.event.inputs.mode != '", fleet_converge_workflow_mode_wire(mode: NamecheapObserve), "'", - " && github.event.inputs.mode != '", fleet_converge_workflow_mode_wire(mode: PrinterWorkflow), "'", -], "") +// Job selection consumes the same mode-to-job authority as the dedicated builders. A second +// hand-maintained exclusion list accidentally ran this credentialed job for controller install. +data fleet_converge_shared_job_if: String = fleet_converge_any_mode_step_if(modes: filter(fleet_converge_workflow_modes, mode => fleet_converge_mode_job_id(mode: mode) == fleet_converge_shared_job_id)) // THE MODE IS MT. COLLINS UNIT 1'S BOOT (its recut is cut 7 of docs/plans/managed-host-untangle.md), // so the job reads that pair's ADMITTED federation and nothing else. An unprovisioned standing diff --git a/dag/gunbc/recurring_failure_mode/workflow_job_selection_diverges_from_mode_assignment.dag b/dag/gunbc/recurring_failure_mode/workflow_job_selection_diverges_from_mode_assignment.dag new file mode 100644 index 00000000000..9ba96b1f723 --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/workflow_job_selection_diverges_from_mode_assignment.dag @@ -0,0 +1,15 @@ +module gunbc.recurring_failure_mode.workflow_job_selection_diverges_from_mode_assignment + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data workflow_job_selection_diverges_from_mode_assignment: RecurringFailureMode = RecurringFailureMode { + identity: "workflow_job_selection_diverges_from_mode_assignment" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607, fleet-converge run 37887389027: microvm_controller_install maps to its dedicated job, but the shared job used a separate exclusion list that did not exclude that mode. Both jobs queued under the same host-mutation concurrency group, so the unnecessary shared job delayed installation and selected its cloud credential prelude for a local-only install.", + "Invalid state: execution conditions disagree with the modeled mode-to-job assignment. A dedicated job exists but does not exclusively own its dispatch. Harm: extra runner occupancy, unrelated credential demands and failure paths, and a blocked dependent operation despite a successful build.", + "Repair: fleet_converge_shared_job_if derives its positive selectors from fleet_converge_mode_job_id over the closed mode roster. The actual shared Job witness admits plan and workspace modes while excluding dedicated controller, dashboard and IAM modes. Deleting that integration restores the duplicate-job specimen and fails the control.", + "Ceiling: structurally derive every job selector from its mode assignment; no independent exclusion list remains for the shared job. Runtime acceptance still reads the dispatched job set rather than treating generated text as evidence that GitHub ran it.", + ], + evidence: [], +} diff --git a/dag/test/claim/workspace_workflow_dispatch_witness_test.dag b/dag/test/claim/workspace_workflow_dispatch_witness_test.dag index c1f627b1890..24bf2a28401 100644 --- a/dag/test/claim/workspace_workflow_dispatch_witness_test.dag +++ b/dag/test/claim/workspace_workflow_dispatch_witness_test.dag @@ -1,9 +1,9 @@ module test.claim.workspace_workflow_dispatch_witness_test -import std.types { Bool } +import std.types { Bool, String } import gunbc.fleet_converge_workflow { WorkspaceAllocationPlan, WorkspaceImageBuild, - fleet_converge_dispatch_inputs, fleet_converge_mode_scope, fleet_converge_mode_job_id, fleet_converge_shared_job_id, + fleet_converge_job, fleet_converge_shared_job_if, fleet_converge_dispatch_inputs, fleet_converge_mode_scope, fleet_converge_mode_job_id, fleet_converge_shared_job_id, fleet_converge_mode_release_product, CompilerPairOnly, fleet_converge_mode_fleet_ssh_key_demand, FleetSshKeyNotConsumed, } @@ -40,3 +40,19 @@ test fn workspace_image_build_preserves_ownership_and_archives_only_the_product( test fn fleet_dispatch_stays_within_github_input_limit() -> Bool { count(fleet_converge_dispatch_inputs) <= 25 } + +// Exercise the actual emitted job. Shared modes must select it positively; dedicated modes +// cannot consume another runner or attempt the shared cloud login as a side effect. +test fn dedicated_installs_do_not_also_run_the_shared_credentialed_job() -> Bool { + match fleet_converge_job().if_condition { + Absent => false + Present { value: condition } => condition == fleet_converge_shared_job_if + && string_contains(s: condition, pattern: "github.event.inputs.mode == 'workspace_image_build'") + && string_contains(s: condition, pattern: "github.event.inputs.mode == 'workspace_allocation_plan'") + && string_contains(s: condition, pattern: "github.event.inputs.mode == 'plan'") + && !string_contains(s: condition, pattern: "microvm_controller_install") + && !string_contains(s: condition, pattern: "dashboard_deploy") + && !string_contains(s: condition, pattern: "gcp_iam_converge") + && !string_contains(s: condition, pattern: " != ") + } +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 6e2c6255e6b..de325670474 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -17,6 +17,7 @@ collected that refusal; its green workflow conclusion is not a boot receipt. | Deployment candidate | Compiler startup creates ignored `rung_drop/roster.dag`, which candidate admission rejects | Exclude that exact compiler-derived file alongside the existing failure-mode roster; preserve sibling and secret-file refusals | | Workflow dispatch | Main declares 28 inputs; GitHub accepts at most 25 | Keep printer selection separate and carry its five operation fields in one `printer_request` JSON input | | Guest image | Builder exists; no image delivery in controller install | Build under fakeroot, archive kernel/rootfs/digest receipt, pin the measured rootfs, verify installed bytes and custody | +| Job selection | Controller install queues the shared credentialed job as well as its dedicated job | Derive shared job selection from the existing mode-to-job authority | | Controller | Installed revision `cfb9ff80652fe0a74093f08cade64f59d9695266` | Install this PR through the existing controller installer | | Slot budget | `fabric-cell-srv1-13.slice` has no installed unit file | Converge the modeled execution-cell boundary; a synthesized systemd slice is not proof | | Cell inventory | Read-only plan `local-2026-10-09T03:18:35Z` refuses foreign `fabric-cell-zp444.slice` | Establish ownership before retiring it; the empty transient slice has no unit file or remaining journal, and is not silently ignored | @@ -62,7 +63,8 @@ empty commissioning prestate. Do not restore the old CI reservation service: disables the agent and agent forwarding. Record the guest identity and a command executed in the guest. 7. Run `workspace_operator_release_wet` with the key and - `GUNBC_WORKSPACE_RELEASE_GENERATION` from the observation. Reconcile with + `GUNBC_WORKSPACE_RELEASE_GENERATION` from `reservation.generation` in the + observation (not the owner-ledger generation). Reconcile with another allocation plan/apply, and verify released state and absent guest. 8. Request a second key, prove the same slot is reused with a new incarnation, and let its short lease expire. Explicitly dispatch allocation plan/apply to @@ -183,3 +185,28 @@ was canceled before installation at `95037360ab1040b9de4efe4671bb53cad473cb34` when the operator selected host-local SSH. The verified image artifact from run `37882239644` remains the image to install. No commissioning or guest allocation has been performed yet. + +Controller installation [37887389027](https://github.com/gunb-ai/gunbc/actions/runs/37887389027) +selects revision `a85cc481af5b07dc89f270925bb55bc6f025c8cd` and image build +`37882239644`. Both srv1 operation checkouts have that revision. The first live +readback before installation found no controller process or queued start, and both +historical root-owned receipts are `ReservationNotObserved` refusals. They are +preserved. Installation and commissioning remain in progress. + +The build job of `37887389027` succeeded. Its source tree is +`5dcdc0feb191c93ac957850344c88796c5651a39`, with request key +`release-bins-38e793db9ef32d629136be09416dabc9da34cdd3139f3541c9e025c146cb75a8`. +The general converge job also queued for this dedicated mode, occupying its shared +host-mutation concurrency key. Both queued jobs were canceled before starting; +there was no installation effect from the workflow. The general job condition now derives +from `fleet_converge_mode_job_id` to remove that second assignment. +The successful build artifact was downloaded and verified on srv1 using the exact +existing installer consumer script and the key recorded by the build job. All +16 binaries passed verification, and the image/kernel hashes still match. The +existing `microvm_controller_install_srv1_wet` entry is running as `ghrunner` +under an operator-local 24 GiB/no-swap scope. No workflow identity was fabricated. + +The workflow-selection repair passes five focused witnesses. Regeneration succeeds, +and an independent check of the emitted YAML finds exactly one consumer job for +each of all 78 dispatch modes. The shared job selector is the only generated +workflow field that changed. All 8,269 source files parse. From edad20b0dede0bf7e3acf585e2723a00754ac423 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 05:51:19 +0000 Subject: [PATCH 09/19] Permit exact installed commissioning-policy readback --- dag/extdeps/tools/gnu_coreutils.dag | 5 ++++ dag/gunbc/executor_privileged_operation.dag | 5 ++++ dag/gunbc/live_deploy/emit.dag | 3 ++- ...erver_cannot_read_its_installed_policy.dag | 15 ++++++++++++ ...pace_commissioning_dispatch_operations.dag | 11 +++++++-- ...workspace_commissioning_executor_input.dag | 24 +++++++++++++++++-- .../workspace_dispatch_fleet_witness_test.dag | 15 +++++++++++- docs/plans/srv1-workspace-vm-bringup.md | 18 ++++++++++++++ 8 files changed, 90 insertions(+), 6 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/installation_observer_cannot_read_its_installed_policy.dag diff --git a/dag/extdeps/tools/gnu_coreutils.dag b/dag/extdeps/tools/gnu_coreutils.dag index 9d301182af4..09918c69883 100644 --- a/dag/extdeps/tools/gnu_coreutils.dag +++ b/dag/extdeps/tools/gnu_coreutils.dag @@ -38,6 +38,11 @@ data min_coreutils_version: VersionConstraint = ">= 8.0" // criterion for climbing to an absolute row, and the counter-example that makes the distinction // load-bearing are recorded once at extdeps.exec.command program_spelling_is_an_observation_claim. fn cat_program() -> ProgramIdentity = uncataloged_program(invocation: "cat") + +// Observed on srv1, 2026-10-09: command -v cat is /usr/bin/cat, root:root mode 0755. +// Privileged policy readback needs an absolute argv shared with its exact sudoers match. +data cat_binary_path: NonEmptyStr = "/usr/bin/cat" + fn mktemp_program() -> ProgramIdentity = uncataloged_program(invocation: "mktemp") fn cp_program() -> ProgramIdentity = uncataloged_program(invocation: "cp") fn printf_program() -> ProgramIdentity = uncataloged_program(invocation: "printf") diff --git a/dag/gunbc/executor_privileged_operation.dag b/dag/gunbc/executor_privileged_operation.dag index 719f871a0b7..e42a26887b6 100644 --- a/dag/gunbc/executor_privileged_operation.dag +++ b/dag/gunbc/executor_privileged_operation.dag @@ -23,6 +23,7 @@ import std.disposition { Disposition, Scaffold, SingleAuthority, RealizationDisp import std.decl_ref { DeclarationRef, WholeDeclaration } import extdeps.sudo.elevation { ElevatedInvocation, sudo_elevate, sudo_elevate_as, sudo_elevate_argv, sudo_elevate_command, sudo_elevate_as_command } import extdeps.exec.command { ArgvCommand } +import extdeps.tools.gnu_coreutils { cat_binary_path, cat_command } import extdeps.tools.env { env_prefixed_argv } import v2.std.orchestration { EnvSet } import gunbc.cli_run_workspace_root_scaffold { gunbc_workspace_root_env_name } @@ -82,6 +83,7 @@ type ExecutorPrivilegedOperation | SystemdRestartUnit { unit: NonEmptyStr } | ReadNftTable { family: NonEmptyStr, table: NonEmptyStr } | ReadNftRuleset + | ReadFile { path: NonEmptyStr } | ReadConntrackExpectations | ReadLegacyXtablesRules | ReadUserspaceConntrackHelpers @@ -196,6 +198,7 @@ fn executor_privileged_operation_argv(op: ExecutorPrivilegedOperation) -> List concat([cat_binary_path], map(cat_command(path: p as String).arguments, word => word as NonEmptyStr)) ReadNftRuleset => concat([ci_runner_sudo_binary_path(b: SudoNft)], nft_list_ruleset_arguments) ReadConntrackExpectations => concat([ci_runner_sudo_binary_path(b: SudoConntrack)], conntrack_list_expectations_arguments) ReadLegacyXtablesRules => concat([ci_runner_sudo_binary_path(b: SudoIptablesLegacySave)], iptables_legacy_save_arguments) @@ -242,6 +245,7 @@ fn executor_privileged_operation_principal(op: ExecutorPrivilegedOperation) -> O SystemdReloadUnit { unit: _ } => RunsAsRoot SystemdRestartUnit { unit: _ } => RunsAsRoot ReadNftTable { family: _, table: _ } => RunsAsRoot + ReadFile { path: _ } => RunsAsRoot ReadNftRuleset => RunsAsRoot ReadConntrackExpectations => RunsAsRoot ReadLegacyXtablesRules => RunsAsRoot @@ -474,6 +478,7 @@ fn executor_privileged_operation_precondition(op: ExecutorPrivilegedOperation) - SystemdReloadUnit { unit: _ } => NoPrecondition SystemdRestartUnit { unit: _ } => NoPrecondition ReadNftTable { family: _, table: _ } => NoPrecondition + ReadFile { path: _ } => NoPrecondition ReadNftRuleset => NoPrecondition ReadConntrackExpectations => NoPrecondition ReadLegacyXtablesRules => NoPrecondition diff --git a/dag/gunbc/live_deploy/emit.dag b/dag/gunbc/live_deploy/emit.dag index 008b711c3d4..1fa3094619c 100644 --- a/dag/gunbc/live_deploy/emit.dag +++ b/dag/gunbc/live_deploy/emit.dag @@ -3555,7 +3555,8 @@ fn workspace_executor_install_steps(revision: ReleaseRevisionBinding) -> List List { let stage = "workspace-commissioning-sudoers" let pending = concat(workspace_dispatch_grant_path, ".pending") diff --git a/dag/gunbc/recurring_failure_mode/installation_observer_cannot_read_its_installed_policy.dag b/dag/gunbc/recurring_failure_mode/installation_observer_cannot_read_its_installed_policy.dag new file mode 100644 index 00000000000..2013c952dd2 --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/installation_observer_cannot_read_its_installed_policy.dag @@ -0,0 +1,15 @@ +module gunbc.recurring_failure_mode.installation_observer_cannot_read_its_installed_policy + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data installation_observer_cannot_read_its_installed_policy: RecurringFailureMode = RecurringFailureMode { + identity: "installation_observer_cannot_read_its_installed_policy" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607: microvm_controller_install_srv1_wet installed release a85cc481af5 on srv1, then refused exact adapter readback. The installed sudoers policy was correctly root:root 0440, but workspace_executor_installed_adapter_matches attempted an ordinary Filesystem.Read as ghrunner. The observer had neither filesystem read permission nor an exact privileged read grant. No VM was commissioned by this installation.", + "Earliest unjustified boundary: installation establishes restricted policy custody, while the adapter observer assumes its caller can read every installed artifact directly. Correct installation makes that assumption false. This differs from demanding an output grant before installation: the read runs after installation, but the installed capability set never provided it.", + "Repair: workspace_dispatch_grant_read_operation names the single policy path. Both its sudoers row and the actual elevated read derive from that operation. Root custody and exact content equality still gate acceptance; unreadable, absent or mismatched policy refuses. Neither file permissions nor access to the protected-state credential is widened.", + "Guarantee: the operation and permission cannot disagree about argv because they share one value; installation_readback_grants_only_the_exact_policy_file executes the grant producer and discriminates this exact path from a wildcard or credential read. Actual account privileges and filesystem custody remain external facts, verified by the installed adapter readback as the job account. Live acceptance is required; the pure witness alone cannot prove sudo admitted that account.", + ], + evidence: [], +} diff --git a/dag/gunbc/workspace_commissioning_dispatch_operations.dag b/dag/gunbc/workspace_commissioning_dispatch_operations.dag index 18e3f86d273..218b660aae4 100644 --- a/dag/gunbc/workspace_commissioning_dispatch_operations.dag +++ b/dag/gunbc/workspace_commissioning_dispatch_operations.dag @@ -1,7 +1,7 @@ module gunbc.workspace_commissioning_dispatch_operations import std.types { String, NonEmptyStr, List } -import gunbc.executor_privileged_operation { ExecutorPrivilegedOperation, RunGunbcEntryAs, executor_privileged_operation_sudoers_line } +import gunbc.executor_privileged_operation { ExecutorPrivilegedOperation, RunGunbcEntryAs, ReadFile, executor_privileged_operation_sudoers_line } import gunbc.runner_microvm_slot_unit { microvm_controller_gunbc_binary, microvm_controller_owner } import gunbc.fleet_generation_execution { fleet_generation_writer } import gunbc.fleet_posix_accounts { fleet_posix_operator_user } @@ -16,8 +16,15 @@ fn workspace_dispatch_operation(release_dir: NonEmptyStr, function: NonEmptyStr) data workspace_dispatch_grant_path: String = "/etc/sudoers.d/gunbc-workspace-commissioning" data workspace_dispatch_functions: List = ["workspace_dispatch_plan_cli", "workspace_dispatch_stage_cli", "workspace_dispatch_complete_cli", "workspace_dispatch_offer_cli", "workspace_dispatch_allocation_plan_cli", "workspace_dispatch_allocation_readback_cli"] +// The grant file remains root:root 0440. Readers may inspect this exact public policy, never +// another file; the command and its sudoers row are projections of this single operation. +fn workspace_dispatch_grant_read_operation() -> ExecutorPrivilegedOperation { + ReadFile { path: workspace_dispatch_grant_path as NonEmptyStr } +} + fn workspace_dispatch_grant_content(release_dir: NonEmptyStr) -> String { join(flat_map([fleet_generation_writer(), fleet_posix_operator_user.name], grantee => map(workspace_dispatch_functions, function => executor_privileged_operation_sudoers_line(grantee: grantee, - op: workspace_dispatch_operation(release_dir: release_dir, function: function)) as String)), "\n") + op: workspace_dispatch_operation(release_dir: release_dir, function: function)) as String) + + [executor_privileged_operation_sudoers_line(grantee: grantee, op: workspace_dispatch_grant_read_operation()) as String]), "\n") } diff --git a/dag/gunbc/workspace_commissioning_executor_input.dag b/dag/gunbc/workspace_commissioning_executor_input.dag index 368b8ac28f5..bbe9dd3f110 100644 --- a/dag/gunbc/workspace_commissioning_executor_input.dag +++ b/dag/gunbc/workspace_commissioning_executor_input.dag @@ -2,7 +2,12 @@ module gunbc.workspace_commissioning_executor_input import gunbc.workspace_commissioning_input_store { workspace_commissioning_input_read, WorkspaceCommissioningInputPresent, WorkspaceCommissioningInputRefused } -import gunbc.workspace_commissioning_dispatch_operations { workspace_dispatch_grant_path, workspace_dispatch_grant_content } +import gunbc.workspace_commissioning_dispatch_operations { workspace_dispatch_grant_path, workspace_dispatch_grant_content, workspace_dispatch_grant_read_operation } + +import gunbc.executor_privileged_operation { executor_privileged_operation_elevated_command } +import gunbc.command_runner { run_shell_command_capture } +import extdeps.exec.command { LocalExec } +import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } import std.types { String, NonEmptyStr, CommitSha, Bool } import std.algebra { trim } @@ -89,6 +94,21 @@ fn workspace_executor_owned_text_equals(path: String, expected: String) -> Bool } } +// Installer readback runs as the job user. Sudoers custody is checked before using the exact +// read-only grant; changing permissions or substituting a successful install is not a readback. +fn workspace_executor_grant_matches(expected: String) -> Bool { + match read_store_ownership(path: workspace_dispatch_grant_path) { + OwnershipRead { ownership: StoreRootOwned } => match run_shell_command_capture( + command: executor_privileged_operation_elevated_command(op: workspace_dispatch_grant_read_operation()), transport: LocalExec) { + ProcessOutputPresent { text } => text == expected + ProcessOutputAbsent => false + ProcessRefused { exit_code: _, stderr: _ } => false + } + OwnershipRead { ownership: StoreNotRootOwned { owner_uid: _, mode: _ } } => false + OwnershipUnread { reason: _ } => false + } +} + fn workspace_executor_installed_adapter_matches(release_dir: String, slot: NonEmptyStr) -> Bool { let unit = workspace_executor_unit_instance_name(slot: slot) let fragment = systemd.Systemctl.ShowProperty(unit: unit, property: "FragmentPath") @@ -97,7 +117,7 @@ fn workspace_executor_installed_adapter_matches(release_dir: String, slot: NonEm fragment.success && trim(s: fragment.value) == (workspace_executor_unit_path() as String) && dropins.success && trim(s: dropins.value) == "" && reload.success && trim(s: reload.value) == "no" - && workspace_executor_owned_text_equals(path: workspace_dispatch_grant_path, expected: concat(workspace_dispatch_grant_content(release_dir: release_dir as NonEmptyStr), "\n")) + && workspace_executor_grant_matches(expected: concat(workspace_dispatch_grant_content(release_dir: release_dir as NonEmptyStr), "\n")) && workspace_executor_owned_text_equals(path: workspace_executor_unit_path() as String, expected: workspace_executor_unit_text(release_dir: release_dir, fleet_lock: fleet_generation_lock, wait_bound: workspace_executor_wait_bound)) && workspace_executor_owned_text_equals(path: join([release_dir, "/", workspace_executor_script_name], ""), expected: workspace_commissioning_executor_script()) diff --git a/dag/test/claim/workspace_dispatch_fleet_witness_test.dag b/dag/test/claim/workspace_dispatch_fleet_witness_test.dag index 4d0063275df..e5117f79142 100644 --- a/dag/test/claim/workspace_dispatch_fleet_witness_test.dag +++ b/dag/test/claim/workspace_dispatch_fleet_witness_test.dag @@ -9,9 +9,10 @@ import gunbc.fleet_converge_plan { fleet_converge_scope_wire } import gunbc.workspace_commissioning_executor_plan { workspace_executor_scope_wire } -import gunbc.workspace_commissioning_dispatch_operations { workspace_dispatch_grant_content } +import gunbc.workspace_commissioning_dispatch_operations { workspace_dispatch_grant_content, workspace_dispatch_grant_read_operation } import gunbc.workspace_commissioning_dispatch_transport { workspace_dispatch_captured_command } import gunbc.fleet_generation_execution { fleet_generation_writer } +import gunbc.executor_privileged_operation { executor_privileged_operation_argv, executor_privileged_operation_sudoers_line } fn dispatch_grant_fixture() -> String { @@ -46,3 +47,15 @@ test fn dispatch_uses_fleet_lock_owner_without_reopening_artifacts() -> Bool { && string_contains(s: operator, pattern: "'captured input'") && !string_contains(s: operator, pattern: "apply.sh") } + +// A job can read exactly its installation policy, without gaining a general root file reader. +test fn installation_readback_grants_only_the_exact_policy_file() -> Bool { + let op = workspace_dispatch_grant_read_operation() + let argv = executor_privileged_operation_argv(op: op) + let grant = executor_privileged_operation_sudoers_line(grantee: fleet_generation_writer(), op: op) as String + argv == ["/usr/bin/cat", "/etc/sudoers.d/gunbc-workspace-commissioning"] + && string_contains(s: dispatch_grant_fixture(), pattern: grant) + && ends_with(s: grant, suffix: "/usr/bin/cat /etc/sudoers.d/gunbc-workspace-commissioning") + && !string_contains(s: grant, pattern: "*") + && !string_contains(s: grant, pattern: "state-writer-mac-key") +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index de325670474..dd5b35c4b5e 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -210,3 +210,21 @@ The workflow-selection repair passes five focused witnesses. Regeneration succee and an independent check of the emitted YAML finds exactly one consumer job for each of all 78 dispatch modes. The shared job selector is the only generated workflow field that changed. All 8,269 source files parse. + +The local `a85cc481af5` installation completed its mutations but returned +`executor scripts or loaded unit failed exact readback`. The final read attempted +to open root:root `0440` `/etc/sudoers.d/gunbc-workspace-commissioning` as +`ghrunner`; that account has no read permission or read grant. `visudo` accepts the +installed policy. Independent readback matches the compiler artifact's SHA-256 +`16f75c38f7b5a3eb7c3580dbe9f6bae59b5997630d808bd3cc3b39ad49dd9930`, both image pins, +and the root-owned release/readiness/commissioning directories. The loaded slot +unit selects `a85cc481af5` and has no running process. Installation convergence is +not yet claimed: the readback needs an exact privileged read of its own policy. + +The repair derives that read's argv and permission from the same privileged +operation, retaining root:root `0440` custody and exact content comparison. Seven +focused dispatch/allocation witnesses pass, including the exact policy-file grant +and the existing prohibition on workflow-created operator intent. The verified +`a85cc481af5` compiler binary can interpret this DAG-only repair; its build identity +remains `a85cc481af5`, separately from the new installed source revision. It is not +represented as a rebuilt compiler pack for the new tree. From 682c32644e84da04dd4e73d334413057a659dbf3 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 06:42:32 +0000 Subject: [PATCH 10/19] Complete local microVM network setup and exact policy readback --- dag/extdeps/tools/gnu_coreutils.dag | 4 - dag/extdeps/tools/sha256sum.dag | 4 + dag/gunbc/executor_privileged_operation.dag | 11 +- dag/gunbc/live_deploy/emit.dag | 2 + ...ontract_is_met_by_a_trimming_transport.dag | 1 + ...claration_signature_consumer_unplanned.dag | 1 + ...erver_cannot_read_its_installed_policy.dag | 2 +- .../receipt_consumer_without_producer.dag | 1 + .../runner_microvm_lifecycle_realize.dag | 2 +- dag/gunbc/runner/runner_microvm_network.dag | 9 +- .../runner/runner_microvm_network_apply.dag | 204 ++++++++++---- .../runner/runner_microvm_network_observe.dag | 253 ++++++++++++++---- .../runner_microvm_network_receipt_store.dag | 79 ++++++ ...ner_microvm_network_sanitation_observe.dag | 76 ++++++ .../runner/runner_microvm_slot_controller.dag | 20 +- ...pace_commissioning_dispatch_operations.dag | 6 +- ...workspace_commissioning_executor_input.dag | 30 ++- dag/gunbc/workspace_initial_sanitation.dag | 2 +- ...ner_microvm_network_apply_witness_test.dag | 16 +- ..._microvm_network_converge_witness_test.dag | 11 + .../workspace_dispatch_fleet_witness_test.dag | 21 +- ...orkspace_network_producer_witness_test.dag | 14 +- docs/plans/srv1-workspace-vm-bringup.md | 49 +++- 23 files changed, 670 insertions(+), 148 deletions(-) create mode 100644 dag/gunbc/runner/runner_microvm_network_receipt_store.dag create mode 100644 dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag diff --git a/dag/extdeps/tools/gnu_coreutils.dag b/dag/extdeps/tools/gnu_coreutils.dag index 09918c69883..bd18e6cf946 100644 --- a/dag/extdeps/tools/gnu_coreutils.dag +++ b/dag/extdeps/tools/gnu_coreutils.dag @@ -39,10 +39,6 @@ data min_coreutils_version: VersionConstraint = ">= 8.0" // load-bearing are recorded once at extdeps.exec.command program_spelling_is_an_observation_claim. fn cat_program() -> ProgramIdentity = uncataloged_program(invocation: "cat") -// Observed on srv1, 2026-10-09: command -v cat is /usr/bin/cat, root:root mode 0755. -// Privileged policy readback needs an absolute argv shared with its exact sudoers match. -data cat_binary_path: NonEmptyStr = "/usr/bin/cat" - fn mktemp_program() -> ProgramIdentity = uncataloged_program(invocation: "mktemp") fn cp_program() -> ProgramIdentity = uncataloged_program(invocation: "cp") fn printf_program() -> ProgramIdentity = uncataloged_program(invocation: "printf") diff --git a/dag/extdeps/tools/sha256sum.dag b/dag/extdeps/tools/sha256sum.dag index 3f308663bf9..f7d52c1c85b 100644 --- a/dag/extdeps/tools/sha256sum.dag +++ b/dag/extdeps/tools/sha256sum.dag @@ -43,6 +43,10 @@ data sha256sum_cli_tool: CliTool = CliTool { installable_via: [SourceCoreutils], } +// Observed on srv1, 2026-10-09: /usr/bin/sha256sum is root:root 0755. Exact sudoers +// digest-read grants require this resolved path, shared by their executor. +data sha256sum_binary_path: NonEmptyStr = "/usr/bin/sha256sum" + // DigestFile DIGESTS a file rather than verifying it against one a caller already has. CheckFile // answers "does this file match this digest", which cannot be used to LEARN a digest: a caller that // wants to record which bytes it actually ran has nothing to compare against yet. coreutils prints diff --git a/dag/gunbc/executor_privileged_operation.dag b/dag/gunbc/executor_privileged_operation.dag index e42a26887b6..8773ed7e5b0 100644 --- a/dag/gunbc/executor_privileged_operation.dag +++ b/dag/gunbc/executor_privileged_operation.dag @@ -23,7 +23,8 @@ import std.disposition { Disposition, Scaffold, SingleAuthority, RealizationDisp import std.decl_ref { DeclarationRef, WholeDeclaration } import extdeps.sudo.elevation { ElevatedInvocation, sudo_elevate, sudo_elevate_as, sudo_elevate_argv, sudo_elevate_command, sudo_elevate_as_command } import extdeps.exec.command { ArgvCommand } -import extdeps.tools.gnu_coreutils { cat_binary_path, cat_command } +import extdeps.tools.sha256sum { sha256sum_binary_path } +import extdeps.crypto.hash { sha256sum_file_command } import extdeps.tools.env { env_prefixed_argv } import v2.std.orchestration { EnvSet } import gunbc.cli_run_workspace_root_scaffold { gunbc_workspace_root_env_name } @@ -83,7 +84,7 @@ type ExecutorPrivilegedOperation | SystemdRestartUnit { unit: NonEmptyStr } | ReadNftTable { family: NonEmptyStr, table: NonEmptyStr } | ReadNftRuleset - | ReadFile { path: NonEmptyStr } + | ReadFileSha256 { path: NonEmptyStr } | ReadConntrackExpectations | ReadLegacyXtablesRules | ReadUserspaceConntrackHelpers @@ -198,7 +199,7 @@ fn executor_privileged_operation_argv(op: ExecutorPrivilegedOperation) -> List concat([cat_binary_path], map(cat_command(path: p as String).arguments, word => word as NonEmptyStr)) + ReadFileSha256 { path: p } => concat([sha256sum_binary_path], map(sha256sum_file_command(path: p as String).arguments, word => word as NonEmptyStr)) ReadNftRuleset => concat([ci_runner_sudo_binary_path(b: SudoNft)], nft_list_ruleset_arguments) ReadConntrackExpectations => concat([ci_runner_sudo_binary_path(b: SudoConntrack)], conntrack_list_expectations_arguments) ReadLegacyXtablesRules => concat([ci_runner_sudo_binary_path(b: SudoIptablesLegacySave)], iptables_legacy_save_arguments) @@ -245,7 +246,7 @@ fn executor_privileged_operation_principal(op: ExecutorPrivilegedOperation) -> O SystemdReloadUnit { unit: _ } => RunsAsRoot SystemdRestartUnit { unit: _ } => RunsAsRoot ReadNftTable { family: _, table: _ } => RunsAsRoot - ReadFile { path: _ } => RunsAsRoot + ReadFileSha256 { path: _ } => RunsAsRoot ReadNftRuleset => RunsAsRoot ReadConntrackExpectations => RunsAsRoot ReadLegacyXtablesRules => RunsAsRoot @@ -478,7 +479,7 @@ fn executor_privileged_operation_precondition(op: ExecutorPrivilegedOperation) - SystemdReloadUnit { unit: _ } => NoPrecondition SystemdRestartUnit { unit: _ } => NoPrecondition ReadNftTable { family: _, table: _ } => NoPrecondition - ReadFile { path: _ } => NoPrecondition + ReadFileSha256 { path: _ } => NoPrecondition ReadNftRuleset => NoPrecondition ReadConntrackExpectations => NoPrecondition ReadLegacyXtablesRules => NoPrecondition diff --git a/dag/gunbc/live_deploy/emit.dag b/dag/gunbc/live_deploy/emit.dag index 1fa3094619c..dcb3a241e27 100644 --- a/dag/gunbc/live_deploy/emit.dag +++ b/dag/gunbc/live_deploy/emit.dag @@ -19,6 +19,7 @@ import gunbc.fleet_generation_execution { fleet_generation_lock } import gunbc.runner_microvm_cell_readiness_location { cell_readiness_store_location } import gunbc.workspace_commissioning_location { workspace_commissioning_store_location } +import gunbc.runner_microvm_network { runner_microvm_converged_receipt_root } import gunbc.live_deploy.belt_cadence { belt_tick_cadence } import gunbc.auth.approval_device_redemption { approval_device_store_directory } @@ -2258,6 +2259,7 @@ fn microvm_controller_install_intent(revision: ReleaseRevisionBinding) -> Pipeli deploy_raw(command: install_d_owned_for_user(user: microvm_controller_owner, path: microvm_slot_controller_receipt_dir)), deploy_raw(command: install_d_owned_for_user(user: microvm_controller_owner, path: cell_readiness_store_location)), deploy_raw(command: install_d_owned_for_user(user: microvm_controller_owner, path: workspace_commissioning_store_location)), + deploy_raw(command: install_d_owned_for_user(user: microvm_controller_owner, path: runner_microvm_converged_receipt_root)), ], concat(microvm_slot_unit_write_steps(revision: revision) + workspace_executor_install_steps(revision: revision) + workspace_dispatch_grant_steps(revision: revision), [deploy_effect(inv: systemctl_daemon_reload_effect())]), ), diff --git a/dag/gunbc/recurring_failure_mode/a_termination_contract_is_met_by_a_trimming_transport.dag b/dag/gunbc/recurring_failure_mode/a_termination_contract_is_met_by_a_trimming_transport.dag index 7966d6aaa40..448d0652e15 100644 --- a/dag/gunbc/recurring_failure_mode/a_termination_contract_is_met_by_a_trimming_transport.dag +++ b/dag/gunbc/recurring_failure_mode/a_termination_contract_is_met_by_a_trimming_transport.dag @@ -8,6 +8,7 @@ data a_termination_contract_is_met_by_a_trimming_transport: RecurringFailureMode identity: "a_termination_contract_is_met_by_a_trimming_transport" as NonEmptyStr, receipts: [ + "2026-10-09, PR #13607: srv1 controller installation at edad20b0ded granted an exact sudo cat of its root-owned 0440 policy, then still refused content equality. The actual file ended in LF; bounded_shell_host_drain CapturedStreamEvidence::from_observation removed that byte with retained_utf8_lossy_trimmed before comparison. The repair measures the installed file with a narrowly granted sha256sum operation and hashes the complete expected bytes through stdin. The digest comparison retains the final-newline distinction without changing capture semantics or accepting trimmed policies. policy_digest_readback_preserves_final_newline_identity discriminates the two byte strings, absent output and refused execution. External privilege and custody still require live readback.", "**a reader whose contract is LINE TERMINATION is fed through a transport whose contract is TRIMMING** (each contract is sound in isolation and they compose to a false reading: the producer really finished, and the reader really reports unfinished.", "THE SHAPE. A reader of a stream being written concurrently must decide what counts as evidence, and the sound answer is TERMINATED LINES ONLY -- a final fragment with no newline is half a statement, and reading it would let a partially written value be read as a whole one. Independently, a capture transport normalizes what it hands back, and trailing-whitespace trimming is the near-universal convention for captured process output, because a caller comparing a captured word against a literal does not want the shell's newline. Neither party is wrong about its own subject. But the trim REMOVES THE EXACT BYTE the reader's contract depends on, so the producer's last line -- which is where terminators live: an END marker, a total, a verdict -- is silently dropped, and the reader reports the state it reserves for a producer that has not finished.", diff --git a/dag/gunbc/recurring_failure_mode/changed_declaration_signature_consumer_unplanned.dag b/dag/gunbc/recurring_failure_mode/changed_declaration_signature_consumer_unplanned.dag index 5fd5f4a880a..019bac22a13 100644 --- a/dag/gunbc/recurring_failure_mode/changed_declaration_signature_consumer_unplanned.dag +++ b/dag/gunbc/recurring_failure_mode/changed_declaration_signature_consumer_unplanned.dag @@ -8,6 +8,7 @@ data changed_declaration_signature_consumer_unplanned: RecurringFailureMode = Re identity: "changed_declaration_signature_consumer_unplanned" as NonEmptyStr, receipts: [ + "2026-10-09, PR #13607 bringup: runner_microvm_network_observe had been replaced by per-attempt sanitation helpers while ci_spec still named its removed runner_microvm_network_observe_wet CLI. The reference is workflow data, outside the declaration-call population described below. Restoring the observer and moving sanitation into runner_microvm_network_sanitation_observe preserves both consumers; workspace_network_producer_witness_test imports the restored production observer so the focused suite prepares its closure. Remaining ceiling: admission must resolve every emitted workflow entry/function pair, including data-carried references, before the required gate accepts it. This repair restores the particular route without claiming that broader gate exists.", "**a declaration's INTERFACE changes -- a product field retyped, an alias re-branded, a function parameter or result retyped, a constructor payload changed, a declaration removed -- and the untouched modules that READ it stay outside the required floor's prepared subject, so main admits an importer that no longer typechecks** (THE INVALID STATE is a merged tree holding a module that refuses under Strict preparation while every required lane that ran on the merging change reported green. The floor prepares the gate closure plus what the diff names: the touched modules and their DEPENDENCY closure. A reader of the changed declaration is a REVERSE dependent with an empty diff, so no diff-keyed selector sees it and it is never prepared.)", "THE HARM is deferred and misattributed. The broken reader sits on main until some LATER, UNRELATED change happens to pull it into its own subject -- and that change's author is the one who meets the refusal, in a file they did not touch, caused by a change they did not make. Every PR in between merges over a red it cannot see. It is the ordinary compiler floor (DESIGN section 4b: values inhabit declared types, fields exist) failing silently on the merge path, which is a below-baseline regression, not a missing differentiator.", diff --git a/dag/gunbc/recurring_failure_mode/installation_observer_cannot_read_its_installed_policy.dag b/dag/gunbc/recurring_failure_mode/installation_observer_cannot_read_its_installed_policy.dag index 2013c952dd2..10d9005d30d 100644 --- a/dag/gunbc/recurring_failure_mode/installation_observer_cannot_read_its_installed_policy.dag +++ b/dag/gunbc/recurring_failure_mode/installation_observer_cannot_read_its_installed_policy.dag @@ -8,7 +8,7 @@ data installation_observer_cannot_read_its_installed_policy: RecurringFailureMod receipts: [ "2026-10-09, PR #13607: microvm_controller_install_srv1_wet installed release a85cc481af5 on srv1, then refused exact adapter readback. The installed sudoers policy was correctly root:root 0440, but workspace_executor_installed_adapter_matches attempted an ordinary Filesystem.Read as ghrunner. The observer had neither filesystem read permission nor an exact privileged read grant. No VM was commissioned by this installation.", "Earliest unjustified boundary: installation establishes restricted policy custody, while the adapter observer assumes its caller can read every installed artifact directly. Correct installation makes that assumption false. This differs from demanding an output grant before installation: the read runs after installation, but the installed capability set never provided it.", - "Repair: workspace_dispatch_grant_read_operation names the single policy path. Both its sudoers row and the actual elevated read derive from that operation. Root custody and exact content equality still gate acceptance; unreadable, absent or mismatched policy refuses. Neither file permissions nor access to the protected-state credential is widened.", + "Repair: workspace_dispatch_grant_read_operation names a SHA-256 measurement of the single policy path. Both its sudoers row and the actual elevated measurement derive from that operation. Root custody and equality with the digest of the complete expected bytes still gate acceptance; unreadable, absent or mismatched policy refuses. Neither file permissions nor access to the protected-state credential is widened. Raw cat readback was replaced after its normalized stdout lost the final LF, recorded in a_termination_contract_is_met_by_a_trimming_transport.", "Guarantee: the operation and permission cannot disagree about argv because they share one value; installation_readback_grants_only_the_exact_policy_file executes the grant producer and discriminates this exact path from a wildcard or credential read. Actual account privileges and filesystem custody remain external facts, verified by the installed adapter readback as the job account. Live acceptance is required; the pure witness alone cannot prove sudo admitted that account.", ], evidence: [], diff --git a/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag b/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag index f3a1eaf186b..d3e1d468a23 100644 --- a/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag +++ b/dag/gunbc/recurring_failure_mode/receipt_consumer_without_producer.dag @@ -6,6 +6,7 @@ import gunbc.recurring_failure_mode { RecurringFailureMode } data receipt_consumer_without_producer: RecurringFailureMode = RecurringFailureMode { identity: "receipt_consumer_without_producer" as NonEmptyStr, receipts: [ + "2026-10-09, PR #13607: srv1 commissioning refused the absent /var/lib/gunbc/microvm-network/converged-slot-network.txt. The earlier observer only wrote a job artifact; no production route published the root-owned receipt the controller consumes. Repair: the installer creates the root-owned receipt directory, and runner_microvm_network_initialize_local_wet binds the local administrator, host and clean source revision, observes all existing convergence predicates, and atomically creates the receipt. A retry observes the same stored generation anew and compares it without replacement; mismatched or unreadable state refuses. Publication admits only this live observer, never caller-supplied receipt text. External network truth remains a live acceptance obligation; no unit test alone establishes it.", "2026-10-09 srv1 workspace image run 37878358778, PR #13607: the compiler pair built successfully, then release-bins packing refused BuildDiagnosticsMissing because the product requires gunbc-floor-cmd.log but fleet's pair build dispatch ran the raw cargo line without the floor's capture wrapper. The pack's refusal was correct. Reusing wrap_command_with_floor_attempt_receipt at that producer captures the required diagnostics and preserves compiler failure through tee; no empty log is synthesized and no pack member is dropped.", "2026-10-09 srv1 workspace bringup, PR #13607: allocation offer admission and the controller require a gateway forwarding observation no older than 60 seconds, but workspace_refresh_gateway_wet had no production caller. The observer existed and its decoder was tested, while every real offer necessarily refused an absent or stale receipt. A bounded foreground operator refresher was initially added, then removed when the operator selected direct SSH from srv1. That route does not exercise gateway forwarding and must not require its receipt: workspace_ssh_access now carries distinct host-local and gateway routes, and each consumer requires only the evidence for its enrolled route. Gateway clients retain their forwarding-proof requirement. Host-local acceptance still requires the invocation-bound guest host key and authenticated guest SSH; a fresh host observation alone is not a login receipt.", "PR #12421 review 5331195485: gunbc_ci_namecheap_observe_invoke supplied receipt_rel as though it captured execution output, but gunbc_invoke_receipt_steps only read an existing file. A successful observer therefore failed the shell step and suppressed its artifact upload. Invalid state: an emitted receipt read with no producer. Harm: successful observation becomes workflow failure. Discriminating facts: execute emitted shell in a clean root with only the declared output; a planted unmatched read must fail. Mitigation: writer, console display and uploader derive the path from one declaration. Ceiling: workflow construction rejects a receipt consumer without a producer edge; next trigger: that consumed workflow admission relation.", diff --git a/dag/gunbc/runner/runner_microvm_lifecycle_realize.dag b/dag/gunbc/runner/runner_microvm_lifecycle_realize.dag index 32513bd0fb5..72c433fcf26 100644 --- a/dag/gunbc/runner/runner_microvm_lifecycle_realize.dag +++ b/dag/gunbc/runner/runner_microvm_lifecycle_realize.dag @@ -1,7 +1,7 @@ module gunbc.runner_microvm_lifecycle_realize import extdeps.systemd.journalctl -import gunbc.runner_microvm_network_observe { observe_slot_network_readings } +import gunbc.runner_microvm_network_sanitation_observe { observe_slot_network_readings } import extdeps.virtualization.firecracker { FirecrackerVmConfig, firecracker_vm_config_document } import std.types { String, NonEmptyStr, Bool, List, Int } diff --git a/dag/gunbc/runner/runner_microvm_network.dag b/dag/gunbc/runner/runner_microvm_network.dag index 54d627c85bd..56d32a0655c 100644 --- a/dag/gunbc/runner/runner_microvm_network.dag +++ b/dag/gunbc/runner/runner_microvm_network.dag @@ -644,6 +644,10 @@ fn runner_slot_netdev_text(net: RunnerSlotNetwork, owner: NonEmptyStr) -> String // ConfigureWithoutCarrier IS REQUIRED AND IS NOT A TUNING KNOB. A tap has no carrier until a guest // opens it, so without this networkd leaves the address unassigned and the host has no route to a // guest that has not booted yet -- which is every guest, at the moment the VM is configured. +// systemd v255 link_ipv6_enabled and link_update_ipv6_sysctl enable IPv6 when networkd owns +// an IPv6 link-local policy. Disable that policy as well as the per-TAP sysctl: +// https://github.com/systemd/systemd/blob/v255/src/network/networkd-link.c +// https://github.com/systemd/systemd/blob/v255/src/network/networkd-sysctl.c fn runner_slot_network_file_text(net: RunnerSlotNetwork) -> String { join([ "[Match]\n", @@ -652,6 +656,8 @@ fn runner_slot_network_file_text(net: RunnerSlotNetwork) -> String { "[Network]\n", "Address=", runner_slot_host_cidr(net: net), "\n", "ConfigureWithoutCarrier=yes\n", + "LinkLocalAddressing=no\n", + "IPv6AcceptRA=no\n", ], "") } @@ -740,7 +746,8 @@ fn runner_slot_network_install_operations( // path. A job-user-writable receipt that root believes is the rostered root-equivalence class with // the principals swapped, so the local observe mode's target/ report is a diagnostic and never // this path. -data runner_microvm_converged_receipt_path: String = "/var/lib/gunbc/microvm-network/converged-slot-network.txt" +data runner_microvm_converged_receipt_root: String = "/var/lib/gunbc/microvm-network" +data runner_microvm_converged_receipt_path: String = concat(runner_microvm_converged_receipt_root, "/converged-slot-network.txt") // THE DIRECTORY IS THE AUTHORITY AND THE FILE PATH IS DERIVED FROM IT, so the place the ruleset is // installed and the place the install ensures exists cannot become two facts. Spelling the leaf as diff --git a/dag/gunbc/runner/runner_microvm_network_apply.dag b/dag/gunbc/runner/runner_microvm_network_apply.dag index 59921cc29d6..b11325d6c66 100644 --- a/dag/gunbc/runner/runner_microvm_network_apply.dag +++ b/dag/gunbc/runner/runner_microvm_network_apply.dag @@ -2,6 +2,12 @@ module gunbc.runner_microvm_network_apply import std.types { String, NonEmptyStr, List, Bool, Int, FilePath } import std.algebra { trim } +import extdeps.exec.command { ArgvCommand, argv_words, LocalExec } +import extdeps.sudo.elevation { sudo_elevate_command } +import gunbc.command_runner { run_shell_command_observe_outcome } +import extdeps.shell.exec { ProcessRan, ProcessNotSpawned } +import gunbc.live_deploy.candidate { CandidateCheckoutObservation, CandidateCheckoutClean, CandidateCheckoutDirty, CandidateCheckoutUnobservable, observe_candidate_checkout } +import gunbc.hostname_read { hostname_short_read, HostnameShortCaptured, HostnameShortCaptureRefused } import std.process { ProcessExit, ExitSuccess, exit_failure } import std.content_hash { Sha256Digest, sha256_hex_digest } import extdeps.filesystem.filesystem_io { Filesystem } @@ -25,7 +31,7 @@ import gunbc.typed_argv_exec { typed_argv_exec_over_fleet_ssh, TypedArgvExecConv import gunbc.typed_remote_file_write { RemoteFileMode, RemoteFileModeOwnerReadWrite, RemoteFileViaNonInteractiveSudo, TypedRemoteFileConverged, TypedRemoteFileConvergenceRefused, - typed_remote_file_write_seal, converge_typed_remote_file, + typed_remote_file_write_seal, converge_typed_remote_file, TypedRemoteFileWriteAdmissionRefused, TypedRemoteFileWriteAdmitted, } import gunbc.runner_host_deploy { RunnerHostDeploy, runner_host_deploy_for, runner_slot_identities_of, @@ -40,15 +46,16 @@ import gunbc.runner_host_file_converge { import gunbc.build_cache_instance { RunnerSlotIdentity } import extdeps.tools { CliTool, apt_package_of } import gunbc.host_cli_dependency { microvm_network_host_cli_requirements } -import gunbc.host_effect { HostEffectTransport, FleetSsh } +import gunbc.host_effect { HostEffectTransport, FleetSsh, LocalShell, SshShell, EmitArtifactThenThinRun } import gunbc.host_effect_realize { Srv3HostPredicate, Srv3PredicateHolds, Srv3PredicateFails, Srv3PredicateUnobserved, srv3_ensure_apt_tool, + srv3_read_token, Srv3EffectiveUserId, Srv3TokenObserved, Srv3TokenAbsent, Srv3TokenUnobserved, } -import gunbc.executor_privileged_operation { ExecutorPrivilegedOperation, EnsureOwnedDirectory, executor_privileged_operation_argv } +import gunbc.executor_privileged_operation { ExecutorPrivilegedOperation, EnsureOwnedDirectory, executor_privileged_operation_argv, executor_privileged_operation_elevated_command } import gunbc.runner_microvm_network { runner_microvm_shared_host_grain, runner_microvm_nft_read_operation, @@ -220,13 +227,23 @@ type ApplyLeg fn run_admin_leg( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, - argv: List, + transport: HostEffectTransport, + command: ArgvCommand, ) -> ApplyLeg { - match typed_argv_exec_over_fleet_ssh(target: subject.target, context: context, argv: argv) { - TypedArgvExecRefused { reason: why } => ApplyLegRefused { reason: why } - TypedArgvExecConverged { result: r } => ApplyLegRan { exit_code: r.exit_code, stdout: r.stdout, stderr: r.stderr } + match transport { + FleetSsh { target, context } => match typed_argv_exec_over_fleet_ssh(target: target, context: context, argv: argv_words(command: command)) { + TypedArgvExecRefused { reason: why } => ApplyLegRefused { reason: why } + TypedArgvExecConverged { result: r } => ApplyLegRan { exit_code: r.exit_code, stdout: r.stdout, stderr: r.stderr } + } + LocalShell => match run_shell_command_observe_outcome(command: command, transport: LocalExec) { + ProcessNotSpawned { program: _, cause } => ApplyLegRefused { reason: cause as String } + ProcessRan { observation: run } => { + ApplyLegRan { exit_code: run.exit_code, stdout: run.stdout, stderr: run.stderr } + } + } + SshShell { ssh_host: _ } => ApplyLegRefused { reason: "network administration requires a verified fleet session or the local administrator" } + EmitArtifactThenThinRun { bootstrap: _, invocation: _ } => ApplyLegRefused { reason: "network administration cannot use artifact-only transport" } } } @@ -236,10 +253,10 @@ type AdministratorPrivilege fn observe_administrator_privilege( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, ) -> AdministratorPrivilege { - match run_admin_leg(subject: subject, context: context, argv: elevated_argv(command: ["true"])) { + match run_admin_leg(subject: subject, transport: transport, command: sudo_elevate_command(command: ["true"])) { ApplyLegRefused { reason: why } => AdministratorPrivilegeMissing { reason: why } ApplyLegRan { exit_code: c, stdout: _, stderr: e } => if c == 0 { @@ -264,17 +281,22 @@ fn observe_administrator_privilege( // produced the defect. Split out as its own row so a claim can discriminate the sudo prefix without // a host. fn runner_microvm_root_only_probe_argv(paths: List, kind: String) -> List { - elevated_argv(command: runner_host_root_only_find_argv(paths: paths, kind: kind)) + argv_words(command: runner_microvm_root_only_probe_command(paths: paths, kind: kind)) +} + +fn runner_microvm_root_only_probe_command(paths: List, kind: String) -> ArgvCommand { + sudo_elevate_command(command: runner_host_root_only_find_argv(paths: paths, kind: kind)) } fn observe_paths_root_only( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, paths: List, kind: String, ) -> RunnerHostPathMetadata { - match run_admin_leg(subject: subject, context: context, argv: runner_microvm_root_only_probe_argv(paths: paths, kind: kind)) { + match run_admin_leg(subject: subject, transport: transport, + command: runner_microvm_root_only_probe_command(paths: paths, kind: kind)) { ApplyLegRefused { reason: why } => PathsUnobservable { reason: why } ApplyLegRan { exit_code: c, stdout: o, stderr: e } => classify_root_only_paths(requested: paths, exit_code: c, stdout: o, stderr: e) } @@ -293,14 +315,14 @@ fn staging_dir_ensure_operation() -> ExecutorPrivilegedOperation { fn run_operation( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, op: ExecutorPrivilegedOperation, ) -> ApplyLeg { run_admin_leg( subject: subject, - context: context, - argv: elevated_argv(command: map(executor_privileged_operation_argv(op: op), w => w as String)), + transport: transport, + command: executor_privileged_operation_elevated_command(op: op), ) } @@ -346,7 +368,7 @@ type OperationSequence { fn run_operations_until_failure( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, ops: List, ) -> List { @@ -360,7 +382,7 @@ fn run_operations_until_failure( { let r = OperationResult { argv: elevated_argv(command: map(executor_privileged_operation_argv(op: op), w => w as String)), - leg: run_operation(subject: subject, context: context, op: op), + leg: run_operation(subject: subject, transport: transport, op: op), } OperationSequence { still_running: operation_result_ok(r: r), @@ -401,25 +423,36 @@ fn staged_write_wire(r: StagedWriteResult) -> String { // second matters less for these files than for a key, and costs nothing. fn write_staged_file( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, file: StagedFile, ) -> StagedWriteResult { StagedWriteResult { path: file.path, - outcome: match converge_typed_remote_file( - target: subject.target, - context: context, - admission: typed_remote_file_write_seal( - path: file.path as FilePath, - content: file.content, - allowed_scope: runner_microvm_admin_staging_scope, - ), - mode: RemoteFileModeOwnerReadWrite, - privilege: RemoteFileViaNonInteractiveSudo, - ) { - TypedRemoteFileConvergenceRefused { step: st, reason: why } => StagedWriteRefused { step: st, reason: why } - TypedRemoteFileConverged { converged: _ } => StagedWritten + outcome: match transport { + FleetSsh { target, context } => match converge_typed_remote_file( + target: target, context: context, + admission: typed_remote_file_write_seal(path: file.path as FilePath, + content: file.content, allowed_scope: runner_microvm_admin_staging_scope), + mode: RemoteFileModeOwnerReadWrite, privilege: RemoteFileViaNonInteractiveSudo) { + TypedRemoteFileConvergenceRefused { step: st, reason: why } => StagedWriteRefused { step: st, reason: why } + TypedRemoteFileConverged { converged: _ } => StagedWritten + } + LocalShell => match typed_remote_file_write_seal(path: file.path as FilePath, + content: file.content, allowed_scope: runner_microvm_admin_staging_scope) { + TypedRemoteFileWriteAdmissionRefused { cause } => StagedWriteRefused { step: "admission", reason: cause as String } + TypedRemoteFileWriteAdmitted { write: _ } => { + let written = Filesystem.WriteOwnerOnly(path: file.path as String, content: file.content) + if !written.success { StagedWriteRefused { step: "local root stage", reason: written.error } } + else { + let read = Filesystem.Read(path: file.path as String) + if read.success && read.content == file.content { StagedWritten } + else { StagedWriteRefused { step: "local root readback", reason: "staged bytes differ or cannot be read" } } + } + } + } + SshShell { ssh_host: _ } => StagedWriteRefused { step: "transport", reason: "unanchored SSH is not an administrator transport" } + EmitArtifactThenThinRun { bootstrap: _, invocation: _ } => StagedWriteRefused { step: "transport", reason: "artifact-only transport cannot write host files" } }, } } @@ -454,10 +487,10 @@ fn listing_digest(d: Sha256FileDigest, listing: String) -> RulesetReadback { // which refuses; it does not fall back to the rendered text. fn read_back_ruleset( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, ) -> RulesetReadback { - match run_operation(subject: subject, context: context, op: runner_microvm_nft_read_operation()) { + match run_operation(subject: subject, transport: transport, op: runner_microvm_nft_read_operation()) { ApplyLegRefused { reason: why } => RulesetNotListed { reason: why } ApplyLegRan { exit_code: c, stdout: o, stderr: e } => if c != 0 { @@ -483,7 +516,7 @@ type HostToolPrerequisite { verdict: Srv3HostPredicate } -fn ensure_host_tool(tool: CliTool, subject: MicrovmNetworkApplySubject, context: FleetSshExecutionContext) -> HostToolPrerequisite +fn ensure_host_tool(tool: CliTool, subject: MicrovmNetworkApplySubject, transport: HostEffectTransport) -> HostToolPrerequisite uses net: Network { match apt_package_of(tool: tool) { @@ -501,7 +534,7 @@ fn ensure_host_tool(tool: CliTool, subject: MicrovmNetworkApplySubject, context: tool: tool, package: p, require_version_probe: false, - transport: FleetSsh { target: subject.target, context: context }, + transport: transport, ), } } @@ -596,10 +629,10 @@ fn apply_run_text(run: MicrovmNetworkApplyRun) -> String { // optional: without it the run could report only that the operations exited zero. fn run_microvm_network_apply( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, ) -> MicrovmNetworkApplyRun { - let privilege = observe_administrator_privilege(subject: subject, context: context) + let privilege = observe_administrator_privilege(subject: subject, transport: transport) match privilege { AdministratorPrivilegeMissing { reason: _ } => MicrovmNetworkApplyRun { @@ -613,9 +646,9 @@ fn run_microvm_network_apply( readback: RulesetNotListed { reason: "the administrator holds no non-interactive sudo on this host, so nothing was staged, installed or read back" }, } AdministratorPrivilegeHeld => { - let prerequisites = map(microvm_network_host_cli_requirements, t => ensure_host_tool(tool: t, subject: subject, context: context)) + let prerequisites = map(microvm_network_host_cli_requirements, t => ensure_host_tool(tool: t, subject: subject, transport: transport)) if fold(prerequisites, init: true, f: (acc, r) => acc && host_tool_prerequisite_holds(r: r)) { - run_privileged_microvm_network_apply(subject: subject, context: context, privilege: privilege, prerequisites: prerequisites) + run_privileged_microvm_network_apply(subject: subject, transport: transport, privilege: privilege, prerequisites: prerequisites) } else { MicrovmNetworkApplyRun { subject: subject, @@ -653,24 +686,24 @@ fn unstaged_microvm_network_apply_run( fn run_privileged_microvm_network_apply( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, privilege: AdministratorPrivilege, prerequisites: List, ) -> MicrovmNetworkApplyRun { - let ensured = run_operation(subject: subject, context: context, op: staging_dir_ensure_operation()) + let ensured = run_operation(subject: subject, transport: transport, op: staging_dir_ensure_operation()) let dir = match ensured { ApplyLegRefused { reason: why } => PathsUnobservable { reason: concat("the staging directory could not be created: ", why) } ApplyLegRan { exit_code: c, stdout: _, stderr: e } => if c != 0 { PathsUnobservable { reason: join(["the staging directory could not be created; install -d exit=", to_string(c), " ", trim(s: e)], "") } } else { - observe_paths_root_only(subject: subject, context: context, paths: staging_ancestor_paths(), kind: "d") + observe_paths_root_only(subject: subject, transport: transport, paths: staging_ancestor_paths(), kind: "d") } } match dir { PathsRootOnly => - run_staged_microvm_network_apply(subject: subject, context: context, privilege: privilege, prerequisites: prerequisites, dir: dir) + run_staged_microvm_network_apply(subject: subject, transport: transport, privilege: privilege, prerequisites: prerequisites, dir: dir) PathsNotRootOnly { paths: _, detail: _ } => unstaged_microvm_network_apply_run(subject: subject, privilege: privilege, prerequisites: prerequisites, dir: dir) PathsUnobservable { reason: _ } => unstaged_microvm_network_apply_run(subject: subject, privilege: privilege, prerequisites: prerequisites, dir: dir) PathsNotApplicable => unstaged_microvm_network_apply_run(subject: subject, privilege: privilege, prerequisites: prerequisites, dir: dir) @@ -700,7 +733,7 @@ fn uninstalled_microvm_network_apply_run( fn run_staged_microvm_network_apply( subject: MicrovmNetworkApplySubject, - context: FleetSshExecutionContext, + transport: HostEffectTransport, privilege: AdministratorPrivilege, prerequisites: List, dir: RunnerHostPathMetadata, @@ -711,16 +744,16 @@ fn run_staged_microvm_network_apply( host: runner_microvm_shared_host_grain, tap_owner: subject.tap_owner, ) - let staged = map(files, f => write_staged_file(subject: subject, context: context, file: f)) + let staged = map(files, f => write_staged_file(subject: subject, transport: transport, file: f)) let all_staged = fold(staged, init: true, f: (acc, r) => acc && staged_write_ok(r: r)) let staged_meta = if !all_staged { PathsNotApplicable } else { - observe_paths_root_only(subject: subject, context: context, paths: map(files, f => f.path as String), kind: "f") + observe_paths_root_only(subject: subject, transport: transport, paths: map(files, f => f.path as String), kind: "f") } match staged_meta { PathsRootOnly => { - let ops = run_operations_until_failure(subject: subject, context: context, ops: microvm_network_converge_operations(slots: subject.slots)) + let ops = run_operations_until_failure(subject: subject, transport: transport, ops: microvm_network_converge_operations(slots: subject.slots)) let all_ops = length(ops) > 0 && fold(ops, init: true, f: (acc, r) => acc && operation_result_ok(r: r)) MicrovmNetworkApplyRun { subject: subject, @@ -731,7 +764,7 @@ fn run_staged_microvm_network_apply( staged_root_only: staged_meta, operations: ops, readback: if all_ops { - read_back_ruleset(subject: subject, context: context) + read_back_ruleset(subject: subject, transport: transport) } else { RulesetNotListed { reason: "an install operation did not succeed, so the listing would not describe the declared ruleset" } }, @@ -780,7 +813,7 @@ fn runner_microvm_network_apply_ci_wet() -> ProcessExit match prepare_fleet_ssh_agent_context(attempt_raw: runner_microvm_network_apply_attempt_raw) { FleetSshContextRefused { cause: c } => exit_failure(reason: c) FleetSshContextReady { context: context, receipt: context_receipt } => { - let run = run_microvm_network_apply(subject: subject, context: context) + let run = run_microvm_network_apply(subject: subject, transport: FleetSsh { target: subject.target, context: context }) let body = join([ "runner_microvm_network_apply probed_at=", probed_at_word(reading: clock_now_probed_at()) as String, "\n", context_receipt, @@ -798,3 +831,74 @@ fn runner_microvm_network_apply_ci_wet() -> ProcessExit } } } + + +// The same staged-file and operation folds are available on the subject host. A local caller +// must already be root: this entry adds no grant, consumes no agent and cannot select a peer. +fn microvm_network_local_admission(host: String, observed_host: String, effective_uid: String, + expected_revision: String, checked_out_revision: String) -> MicrovmNetworkApplyAdmission { + if trim(s: host) == "" || trim(s: host) != trim(s: observed_host) { + MicrovmNetworkApplyRefused { cause: "local network administration must name this host" } + } else if trim(s: effective_uid) != "0" { + MicrovmNetworkApplyRefused { cause: "local network administration requires the administrator's root process" } + } else { microvm_network_apply_admission(host: host, expected_revision: expected_revision, + checked_out_revision: checked_out_revision) } +} + +fn microvm_network_local_subject(host: String, expected_revision: String) -> MicrovmNetworkApplyAdmission { + match observe_candidate_checkout() { + CandidateCheckoutClean => microvm_network_local_subject_from_clean_checkout(host: host, expected_revision: expected_revision) + CandidateCheckoutDirty { tracked_changes: _, staged_changes: _, untracked_deployed_paths: _, ignored_deployed_paths: _ } => + MicrovmNetworkApplyRefused { cause: "local network administration requires a clean checkout of the named revision" } + CandidateCheckoutUnobservable { detail } => MicrovmNetworkApplyRefused { cause: detail } + } +} + +fn microvm_network_local_subject_from_clean_checkout(host: String, expected_revision: String) -> MicrovmNetworkApplyAdmission { + match hostname_short_read(transport: LocalShell) { + HostnameShortCaptureRefused { reason } => MicrovmNetworkApplyRefused { cause: reason } + HostnameShortCaptured { value: actual } => { + let candidate = microvm_network_apply_admission(host: host, expected_revision: expected_revision, + checked_out_revision: apply_checked_out_revision()) + match candidate { + MicrovmNetworkApplyRefused { cause } => MicrovmNetworkApplyRefused { cause: cause } + MicrovmNetworkApplyAdmitted { subject } => match srv3_read_token(subject: Srv3EffectiveUserId, transport: LocalShell) { + Srv3TokenUnobserved { reason } => MicrovmNetworkApplyRefused { cause: reason } + Srv3TokenAbsent => + MicrovmNetworkApplyRefused { cause: "local administrator identity could not be read" } + Srv3TokenObserved { value: uid } => microvm_network_local_admission(host: host, observed_host: actual, effective_uid: uid, + expected_revision: expected_revision, checked_out_revision: subject.revision) + } + } + } + } +} + +fn runner_microvm_network_plan_local_wet(host: String, expected_revision: String) -> ProcessExit { + match microvm_network_local_subject(host: host, expected_revision: expected_revision) { + MicrovmNetworkApplyRefused { cause } => exit_failure(reason: cause) + MicrovmNetworkApplyAdmitted { subject } => { + let files = converge_staged_files(slots: subject.slots, host: runner_microvm_shared_host_grain, tap_owner: subject.tap_owner) + let body = join(["host=", host, " revision=", subject.revision, " execution=local-root\n", + join(map(files, file => join(["FILE ", file.path as String, "\n", file.content, "\n"], "")), ""), + join(map(microvm_network_converge_operations(slots: subject.slots), op => + concat(to_string(executor_privileged_operation_argv(op: op)), "\n")), "")], "") + let written = Filesystem.Write(path: "target/microvm-network-local-plan.txt", content: body) + if written.success { ExitSuccess } else { exit_failure(reason: written.error) } + } + } +} + +fn runner_microvm_network_apply_local_wet(host: String, expected_revision: String) -> ProcessExit { + match microvm_network_local_subject(host: host, expected_revision: expected_revision) { + MicrovmNetworkApplyRefused { cause } => exit_failure(reason: cause) + MicrovmNetworkApplyAdmitted { subject } => { + let run = run_microvm_network_apply(subject: subject, transport: LocalShell) + let written = Filesystem.Write(path: runner_microvm_network_apply_receipt_path, + content: concat("execution=local-root\n", apply_run_text(run: run))) + if !written.success { exit_failure(reason: written.error) } + else if apply_run_succeeded(run: run) { ExitSuccess } + else { exit_failure(reason: apply_run_text(run: run)) } + } + } +} diff --git a/dag/gunbc/runner/runner_microvm_network_observe.dag b/dag/gunbc/runner/runner_microvm_network_observe.dag index 11308eec2ff..283cf0b2c57 100644 --- a/dag/gunbc/runner/runner_microvm_network_observe.dag +++ b/dag/gunbc/runner/runner_microvm_network_observe.dag @@ -1,76 +1,215 @@ module gunbc.runner_microvm_network_observe -import std.types { String, NonEmptyStr, Bool } +import std.types { String, NonEmptyStr, List } +import gunbc.clock_read { clock_now_probed_at } +import gunbc.runner_microvm_network_receipt_store { read_network_receipt, publish_initial_network_receipt, NetworkReceiptAbsent, NetworkReceiptPresent, StoredNetworkReceiptUnreadable } +import gunbc.runner_microvm_network_apply { MicrovmNetworkApplySubject, microvm_network_local_subject, MicrovmNetworkApplyAdmitted, MicrovmNetworkApplyRefused } import std.algebra { trim } -import std.resources { Filesystem } -import std.content_hash { sha256_hex_digest } -import extdeps.linux.tun_fdinfo { tun_descriptor_census_observe, TunDescriptorCensus, TunDescriptorCount, TunInterfaceMissing, TunDescriptorUnreadable } -import extdeps.exec.command { ArgvCommand, LocalExec } -import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent } -import extdeps.tools.sha256sum { sha256sum_stdin_digest_via_shell, Sha256FileDigest } -import gunbc.command_runner { run_shell_command_capture } -import gunbc.runner_microvm_network { - SlotNetworkSanitationSubject, SlotNetworkReadings, - TapHolderObservation, TapPresentUnheld, TapHeld, TapMissing, TapHoldersUnreadable, - TapIpv6Standing, TapIpv6Disabled, TapIpv6Enabled, TapIpv6Unread, - ResidueObservation, ResidueAbsent, ResiduePresent, ResidueUnreadable, - RulesetObservation, RulesetListed, RulesetUnreadable, - TapReading, TapIpv6Reading, SlotResidueReading, HostResidueReading, RulesetReading, - runner_slot_network, runner_slot_ipv6_disabled_proc_path, - runner_slot_guest_originated_conntrack_list_command, runner_slot_guest_destined_conntrack_list_command, - runner_slot_guest_neighbour_show_command, runner_host_expectation_list_command, runner_microvm_nft_list_command, +import std.process { ProcessExit, ExitSuccess, exit_failure } +import extdeps.shell.exec { ProcessOutcome, ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } +import extdeps.filesystem.filesystem_io { Filesystem } +import extdeps.tools.mkdir { mkdir_parents_command } +import product.host_identity { HostIdentity } +import gunbc.command_runner { run_shell_commands, run_shell_command_capture, LocalExec } +import gunbc.hostname_read { hostname_short_read, HostnameShortCaptured, HostnameShortCaptureRefused } +import gunbc.host_effect { HostEffectTransport, LocalShell, FleetSsh, SshShell, EmitArtifactThenThinRun } +import gunbc.fleet_ssh_locus { prepare_fleet_ssh_agent_context, FleetSshContextReady, FleetSshContextRefused } +import gunbc.fleet_known_hosts_anchor { FleetSshExecutionContext, SshTarget } +import gunbc.fleet_bootstrap_principal { executor_bootstrap_principal } +import gunbc.typed_argv_exec { typed_argv_exec_over_fleet_ssh, TypedArgvExecConverged, TypedArgvExecRefused } +import gunbc.runner_host_deploy { + RunnerHostDeploy, runner_host_deploy_for, + RunnerHostDeployFound, RunnerHostDeployUnmodeled, RunnerHostDeployRefused, } - -fn observe_tap_holders(tap: NonEmptyStr) -> TapHolderObservation { - tap_holders_reading(census: tun_descriptor_census_observe(name: tap as String)) +import gunbc.runner_host_grants { bootstrap_principal_is_not_the_grantee } +import gunbc.runner_host_file_converge { elevated_argv } +import gunbc.build_cache_instance { RunnerSlotIdentity } +import gunbc.executor_privileged_operation { + ExecutorPrivilegedOperation, ReadNftRuleset, ReadLegacyXtablesRules, ReadUserspaceConntrackHelpers, + ReadConntrackExpectations, executor_privileged_operation_argv, executor_privileged_operation_elevated_command, +} +import gunbc.runner_microvm_network { ConvergeGeneration, runner_microvm_nft_read_operation } +import gunbc.runner_microvm_network_converge { + MicrovmPrivilegedReadings, ConvergedNetwork, ConvergeRefused, + read_slot_wet, read_host_wet, converge_outcome, receipt_text, + bind_requested_host, expected_host_reading, host_binding_refusal_text, + HostBindingHeld, HostBindingRefused, HostBindingUnnamed, + converge_generation_reading, runner_microvm_network_slot_identities_of, + runner_microvm_network_receipt_path, runner_microvm_network_receipt_dir, } +import gunbc.runner_microvm_network_apply { runner_microvm_administrator_target } -fn tap_holders_reading(census: TunDescriptorCensus) -> TapHolderObservation { - match census { - TunDescriptorCount { count: n, exited: _ } => if n == 0 { TapPresentUnheld } else { TapHeld { holders: n } } - TunInterfaceMissing => TapMissing - TunDescriptorUnreadable { cause: _ } => TapHoldersUnreadable { reason: "root TUN descriptor census unavailable or malformed" as NonEmptyStr } +// THE OBSERVE READS AS TWO PRINCIPALS, AND WHICH ONE READS WHAT IS THE POINT. +// +// ConvergedSlotNetwork says what a host IS at a generation, so every fact it carries must be a live +// reading. Five of them -- the installed table's listing (whose digest is the converged digest), the +// whole ruleset, the legacy xtables rules, the userspace conntrack helpers and the conntrack +// expectations -- are readable only by root. The job user this step runs as is granted none of +// them, and must not be: operator decision 2026-09-22 keeps gunbc-ghrunner.sudoers modeled and +// uninstalled, because the principal every CI step runs as holding even readbacks is a grant with +// no need behind it once the administrator can read. So those five are read by the host's +// ADMINISTRATOR, gunbc.fleet_bootstrap_principal executor_bootstrap_principal, over the fleet SSH +// edge -- the same principal and the same target gunbc.runner_microvm_network_apply installs through +// -- and everything an unprivileged principal CAN read (procfs, ip, uname) is still read locally. +// +// The first observe run after the apply landed (fleet-converge run 35853748081) refused with +// `the installed ruleset could not be listed`: the listing was attempted as the job user with no +// grant. That refusal was correct and is the reading this module answers, not one it relaxes. + +// A READING THAT DID NOT RUN IS NOT AN EMPTY READING. A refused edge or a non-zero exit is +// ProcessRefused, so the fold refuses by name; only an exit-zero leg with no output is +// ProcessOutputAbsent, which for conntrack's expectation table is a real, empty reading. +fn admin_read_outcome( + target: SshTarget, + context: FleetSshExecutionContext, + op: ExecutorPrivilegedOperation, +) -> ProcessOutcome +{ + let argv = elevated_argv(command: map(executor_privileged_operation_argv(op: op), w => w as String)) + match typed_argv_exec_over_fleet_ssh(target: target, context: context, argv: argv) { + TypedArgvExecRefused { reason: why } => ProcessRefused { exit_code: -1, stderr: why } + TypedArgvExecConverged { result: r } => + if r.exit_code != 0 { + ProcessRefused { exit_code: r.exit_code, stderr: trim(s: r.stderr) } + } else if r.stdout == "" { + ProcessOutputAbsent + } else { + ProcessOutputPresent { text: r.stdout } + } } } -fn observe_network_residue(command: ArgvCommand) -> ResidueObservation { - match run_shell_command_capture(command: command, transport: LocalExec) { - ProcessOutputPresent { text: body } => if trim(s: body) == "" { ResidueAbsent } - else { ResiduePresent { entries: length(filter(split(s: body, delimiter: "\n"), line => trim(s: line) != "")) } } - ProcessOutputAbsent => ResidueAbsent - other => ResidueUnreadable { reason: "network residue listing failed" as NonEmptyStr } +fn read_privileged_network(transport: HostEffectTransport) -> MicrovmPrivilegedReadings +{ + MicrovmPrivilegedReadings { + table: network_read_outcome(transport: transport, op: runner_microvm_nft_read_operation()), + ruleset: network_read_outcome(transport: transport, op: ReadNftRuleset), + xtables: network_read_outcome(transport: transport, op: ReadLegacyXtablesRules), + userspace: network_read_outcome(transport: transport, op: ReadUserspaceConntrackHelpers), + expectations: network_read_outcome(transport: transport, op: ReadConntrackExpectations), } } -fn observe_network_ruleset() -> RulesetObservation { - match run_shell_command_capture(command: runner_microvm_nft_list_command(), transport: LocalExec) { - ProcessOutputPresent { text: body } => match sha256sum_stdin_digest_via_shell(content: body) { - Sha256FileDigest { digest: hashed } => match sha256_hex_digest(hex: hashed.hex as String) { - Present { value: actual_digest } => RulesetListed { digest: actual_digest } - Absent => RulesetUnreadable { reason: "listed ruleset digest malformed" as NonEmptyStr } +data runner_microvm_network_observe_attempt_raw: String = "runner-microvm-network-observe-ci" + +fn runner_microvm_network_observe_wet() -> ProcessExit +{ + let host_reading = hostname_short_read(transport: LocalShell) + match host_reading { + HostnameShortCaptureRefused { reason: r } => + exit_failure(reason: join(["microvm-network-observe: the host could not name itself: ", r], "")) + HostnameShortCaptured { value: name } => + match bind_requested_host(expected: expected_host_reading(), observed: name) { + HostBindingUnnamed { detail: d } => + exit_failure(reason: host_binding_refusal_text(b: HostBindingUnnamed { detail: d })) + HostBindingRefused { expected: e, observed: o } => + exit_failure(reason: host_binding_refusal_text(b: HostBindingRefused { expected: e, observed: o })) + HostBindingHeld { host: bound } => observe_bound_host_wet(bound: bound) } - other => RulesetUnreadable { reason: "listed ruleset digest unavailable" as NonEmptyStr } + } +} + +fn observe_bound_host_wet(bound: HostIdentity) -> ProcessExit +{ + let generation = converge_generation_reading() + if generation == "" { + exit_failure(reason: "microvm-network-observe: no run identity in the environment, so the receipt would carry a generation nobody can compare against") + } else { + match runner_host_deploy_for(host: bound) { + RunnerHostDeployUnmodeled { host: h } => + exit_failure(reason: join(["microvm-network-observe: ", h as String, " is not a modeled runner host"], "")) + RunnerHostDeployRefused { host: h, cause: c } => + exit_failure(reason: join(["microvm-network-observe: ", h as String, " refuses its deploy row: ", c], "")) + RunnerHostDeployFound { deploy: d } => + observe_admitted_host_wet(deploy: d, host: bound, generation: generation as NonEmptyStr) + } + } +} + +// THE READER OF THE ROOT-ONLY SURFACES MAY NOT BE THE JOB USER, for the reason the apply refuses on: +// reading "as the administrator" when the administrator IS the job user would mean the job user +// holds the reads, which is the grant this module exists not to need. +fn observe_admitted_host_wet(deploy: RunnerHostDeploy, host: HostIdentity, generation: NonEmptyStr) -> ProcessExit +{ + if !bootstrap_principal_is_not_the_grantee(bootstrap: executor_bootstrap_principal, deploy: deploy) { + exit_failure(reason: join(["microvm-network-observe: the administrator principal ", executor_bootstrap_principal.login as String, " is the job user on ", host as String, ", so the root-only readings would be taken by the account CI jobs run as"], "")) + } else { + match prepare_fleet_ssh_agent_context(attempt_raw: runner_microvm_network_observe_attempt_raw) { + FleetSshContextRefused { cause: c } => exit_failure(reason: join(["microvm-network-observe: ", c], "")) + FleetSshContextReady { context: context, receipt: _ } => + record_converge_readback_wet( + slots: runner_microvm_network_slot_identities_of(deploy: deploy), + host: host, + generation: generation, + reads: read_privileged_network(transport: FleetSsh { target: runner_microvm_administrator_target(host: host), context: context }), + ) } - other => RulesetUnreadable { reason: "converged nft table could not be listed" as NonEmptyStr } } } -// Actual host reads, joined by the existing subject and converge receipt. A missing tool or -// denied kernel read is unreadable; no exited process or no-carrier state substitutes for these. -fn observe_slot_network_readings(subject: SlotNetworkSanitationSubject) -> SlotNetworkReadings { - let net = runner_slot_network(slot: subject.subject.slot) - let ipv6 = Filesystem.Read(path: runner_slot_ipv6_disabled_proc_path(net: net)) - let ipv6_standing = if !ipv6.success { TapIpv6Unread { reason: "tap IPv6 policy unreadable" as NonEmptyStr } } - else if trim(s: ipv6.content) == "1" { TapIpv6Disabled } - else if trim(s: ipv6.content) == "0" { TapIpv6Enabled } - else { TapIpv6Unread { reason: "tap IPv6 policy malformed" as NonEmptyStr } } - SlotNetworkReadings { - tap: TapReading { subject: subject.subject, observation: observe_tap_holders(tap: net.tap_device) }, - tap_ipv6: TapIpv6Reading { subject: subject.subject, observation: ipv6_standing }, - guest_originated_conntrack: SlotResidueReading { subject: subject.subject, observation: observe_network_residue(command: runner_slot_guest_originated_conntrack_list_command(net: net)) }, - guest_destined_conntrack: SlotResidueReading { subject: subject.subject, observation: observe_network_residue(command: runner_slot_guest_destined_conntrack_list_command(net: net)) }, - guest_neighbour: SlotResidueReading { subject: subject.subject, observation: observe_network_residue(command: runner_slot_guest_neighbour_show_command(net: net)) }, - expectations: HostResidueReading { host: subject.converged.host, generation: subject.converged.generation, observation: observe_network_residue(command: runner_host_expectation_list_command()) }, - ruleset: RulesetReading { host: subject.converged.host, generation: subject.converged.generation, observation: observe_network_ruleset() }, +fn record_converge_readback_wet( + slots: List, + host: HostIdentity, + generation: NonEmptyStr, + reads: MicrovmPrivilegedReadings, +) -> ProcessExit { + let outcome = converge_outcome( + slots: map(slots, s => read_slot_wet(slot: s)), + h: read_host_wet(host: host, generation: generation as ConvergeGeneration, reads: reads), + ) + let text = receipt_text(o: outcome) + let _ = run_shell_commands(commands: [mkdir_parents_command(path: runner_microvm_network_receipt_dir)], transport: LocalExec) + let w = Filesystem.Write(path: runner_microvm_network_receipt_path, content: text) + if !w.success { + exit_failure(reason: join(["microvm-network-observe: receipt write refused: ", w.error], "")) + } else { + match outcome { + ConvergedNetwork { receipt: _ } => ExitSuccess + ConvergeRefused { refusal: _ } => exit_failure(reason: join(["microvm-network-observe: ", text], "")) + } } } + + +fn network_read_outcome(transport: HostEffectTransport, op: ExecutorPrivilegedOperation) -> ProcessOutcome { + match transport { + FleetSsh { target, context } => admin_read_outcome(target: target, context: context, op: op) + LocalShell => run_shell_command_capture(command: executor_privileged_operation_elevated_command(op: op), transport: LocalExec) + SshShell { ssh_host: _ } => ProcessRefused { exit_code: -1, stderr: "network readback requires the local administrator or a verified fleet session" } + EmitArtifactThenThinRun { bootstrap: _, invocation: _ } => ProcessRefused { exit_code: -1, stderr: "network readback requires a live host" } + } +} + +// An explicit administrator action publishes the missing controller-consumed receipt. The CI +// observer above remains read-only apart from its job artifact. No workflow-supplied text can be +// installed as an authoritative network generation, and this entry never overwrites one. +fn runner_microvm_network_initialize_local_wet(host: String, expected_revision: String) -> ProcessExit { + match microvm_network_local_subject(host: host, expected_revision: expected_revision) { + MicrovmNetworkApplyRefused { cause } => exit_failure(reason: cause) + MicrovmNetworkApplyAdmitted { subject } => match read_network_receipt(candidates: subject.slots) { + StoredNetworkReceiptUnreadable { reason } => exit_failure(reason: reason) + NetworkReceiptPresent { receipt } => initialize_local_network_receipt(subject: subject, generation: receipt.generation as NonEmptyStr) + NetworkReceiptAbsent => match clock_now_probed_at() { + Absent => exit_failure(reason: "network initialization clock is unavailable") + Present { value: now } => initialize_local_network_receipt(subject: subject, + generation: join(["local-", subject.revision, "-", now as String], "") as NonEmptyStr) + } + } + } +} + +fn initialize_local_network_receipt(subject: MicrovmNetworkApplySubject, generation: NonEmptyStr) -> ProcessExit { + let outcome = converge_outcome(slots: map(subject.slots, slot => read_slot_wet(slot: slot)), + h: read_host_wet(host: subject.host, generation: generation as ConvergeGeneration, + reads: read_privileged_network(transport: LocalShell))) + let artifact = Filesystem.Write(path: runner_microvm_network_receipt_path, content: receipt_text(o: outcome)) + if !artifact.success { exit_failure(reason: artifact.error) } + else { match outcome { + ConvergeRefused { refusal: _ } => exit_failure(reason: receipt_text(o: outcome)) + ConvergedNetwork { receipt } => match publish_initial_network_receipt(receipt: receipt, candidates: subject.slots) { + NetworkReceiptPresent { receipt: _ } => ExitSuccess + NetworkReceiptAbsent => exit_failure(reason: "network receipt was not published") + StoredNetworkReceiptUnreadable { reason } => exit_failure(reason: reason) + } + } } +} diff --git a/dag/gunbc/runner/runner_microvm_network_receipt_store.dag b/dag/gunbc/runner/runner_microvm_network_receipt_store.dag new file mode 100644 index 00000000000..597ae85af81 --- /dev/null +++ b/dag/gunbc/runner/runner_microvm_network_receipt_store.dag @@ -0,0 +1,79 @@ +module gunbc.runner_microvm_network_receipt_store + +import std.types { String, NonEmptyStr, Bool, List } +import std.decl_ref { decl_ref } +import gunbc.build_cache_instance { RunnerSlotIdentity } +import gunbc.file_access { path_ancestor_prefixes } +import gunbc.runner_microvm_cell_readiness { StoreRootOwned, StoreNotRootOwned } +import gunbc.runner_microvm_cell_readiness_store { read_store_ownership, OwnershipRead, OwnershipUnread, entry_is_root_authoritative } +import extdeps.filesystem.filesystem_io { + Filesystem, filesystem_exact_read, FilesystemExactPathRead, FilesystemExactPathAbsent, + FilesystemExactPathUnreadable, FilesystemExactPathKindUnrecognized, + filesystem_create_new, FilesystemCreated, FilesystemCreateTargetOccupied, + FilesystemCreateRefused, FilesystemCreateKindUnrecognized, +} +import gunbc.runner_microvm_network { + ConvergedSlotNetwork, runner_microvm_converged_receipt_path, + render_converged_slot_network, parse_converged_slot_network, + ConvergedSlotNetworkParsed, ConvergedSlotNetworkMalformed, +} + +type NetworkReceiptRead + = NetworkReceiptAbsent + | NetworkReceiptPresent { receipt: ConvergedSlotNetwork } + | StoredNetworkReceiptUnreadable { reason: String } + +fn network_receipt_ancestors_owned() -> Bool { + all(["/"] + path_ancestor_prefixes(path: runner_microvm_converged_receipt_path), path => + match read_store_ownership(path: path) { + OwnershipRead { ownership: StoreRootOwned } => true + OwnershipRead { ownership: StoreNotRootOwned { owner_uid: _, mode: _ } } => false + OwnershipUnread { reason: _ } => false + }) +} + +fn read_network_receipt(candidates: List) -> NetworkReceiptRead { + if !network_receipt_ancestors_owned() { StoredNetworkReceiptUnreadable { reason: "network receipt ancestors are absent or outside root custody" } } + else { + let read = Filesystem.Read(path: runner_microvm_converged_receipt_path) + match filesystem_exact_read(path: runner_microvm_converged_receipt_path, content: read.content, + success: read.success, error: read.error, error_kind: read.error_kind) { + FilesystemExactPathAbsent { path: _ } => NetworkReceiptAbsent + FilesystemExactPathUnreadable { path: _, kind: _, error } => StoredNetworkReceiptUnreadable { reason: error } + FilesystemExactPathKindUnrecognized { path: _, observed: _, error } => StoredNetworkReceiptUnreadable { reason: error } + FilesystemExactPathRead { path: _, content } => match entry_is_root_authoritative(path: runner_microvm_converged_receipt_path) { + Present { value: reason } => StoredNetworkReceiptUnreadable { reason: reason as String } + Absent => match parse_converged_slot_network(text: content, candidates: candidates) { + ConvergedSlotNetworkParsed { receipt } => NetworkReceiptPresent { receipt: receipt } + ConvergedSlotNetworkMalformed { cause } => StoredNetworkReceiptUnreadable { reason: to_string(cause) } + } + } + } + } +} + +// Initial publication is atomic and never replaces a generation a controller may already use. +// A rerun verifies the stored generation against fresh host readings before reaching this entry. +fn publish_initial_network_receipt(receipt: ConvergedSlotNetwork, candidates: List) -> NetworkReceiptRead + admit_callers: [decl_ref(module_path: "gunbc.runner_microvm_network_observe", decl_name: "initialize_local_network_receipt")] +{ + if !network_receipt_ancestors_owned() { StoredNetworkReceiptUnreadable { reason: "network receipt store is not installed under root custody" } } + else { + let text = render_converged_slot_network(c: receipt) + let write = Filesystem.WriteCreateNew(path: runner_microvm_converged_receipt_path, content: text) + let published = match filesystem_create_new(path: runner_microvm_converged_receipt_path, + success: write.success, error: write.error, error_kind: write.error_kind) { + FilesystemCreated { path: _ } => true + FilesystemCreateTargetOccupied { path: _ } => true + FilesystemCreateRefused { path: _, kind: _, error: _ } => false + FilesystemCreateKindUnrecognized { path: _, observed: _, error: _ } => false + } + if !published { StoredNetworkReceiptUnreadable { reason: "network receipt publication refused" } } + else { match read_network_receipt(candidates: candidates) { + NetworkReceiptPresent { receipt: actual } => if render_converged_slot_network(c: actual) == text { + NetworkReceiptPresent { receipt: actual } + } else { StoredNetworkReceiptUnreadable { reason: "existing network generation differs; explicit recovery is required" } } + other => other + } } + } +} diff --git a/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag b/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag new file mode 100644 index 00000000000..2fcf3bde4bf --- /dev/null +++ b/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag @@ -0,0 +1,76 @@ +module gunbc.runner_microvm_network_sanitation_observe + +import std.types { String, NonEmptyStr, Bool } +import std.algebra { trim } +import std.resources { Filesystem } +import std.content_hash { sha256_hex_digest } +import extdeps.linux.tun_fdinfo { tun_descriptor_census_observe, TunDescriptorCensus, TunDescriptorCount, TunInterfaceMissing, TunDescriptorUnreadable } +import extdeps.exec.command { ArgvCommand, LocalExec } +import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent } +import extdeps.tools.sha256sum { sha256sum_stdin_digest_via_shell, Sha256FileDigest } +import gunbc.command_runner { run_shell_command_capture } +import gunbc.runner_microvm_network { + SlotNetworkSanitationSubject, SlotNetworkReadings, + TapHolderObservation, TapPresentUnheld, TapHeld, TapMissing, TapHoldersUnreadable, + TapIpv6Standing, TapIpv6Disabled, TapIpv6Enabled, TapIpv6Unread, + ResidueObservation, ResidueAbsent, ResiduePresent, ResidueUnreadable, + RulesetObservation, RulesetListed, RulesetUnreadable, + TapReading, TapIpv6Reading, SlotResidueReading, HostResidueReading, RulesetReading, + runner_slot_network, runner_slot_ipv6_disabled_proc_path, + runner_slot_guest_originated_conntrack_list_command, runner_slot_guest_destined_conntrack_list_command, + runner_slot_guest_neighbour_show_command, runner_host_expectation_list_command, runner_microvm_nft_list_command, +} + +fn observe_tap_holders(tap: NonEmptyStr) -> TapHolderObservation { + tap_holders_reading(census: tun_descriptor_census_observe(name: tap as String)) +} + +fn tap_holders_reading(census: TunDescriptorCensus) -> TapHolderObservation { + match census { + TunDescriptorCount { count: n, exited: _ } => if n == 0 { TapPresentUnheld } else { TapHeld { holders: n } } + TunInterfaceMissing => TapMissing + TunDescriptorUnreadable { cause: _ } => TapHoldersUnreadable { reason: "root TUN descriptor census unavailable or malformed" as NonEmptyStr } + } +} + +fn observe_network_residue(command: ArgvCommand) -> ResidueObservation { + match run_shell_command_capture(command: command, transport: LocalExec) { + ProcessOutputPresent { text: body } => if trim(s: body) == "" { ResidueAbsent } + else { ResiduePresent { entries: length(filter(split(s: body, delimiter: "\n"), line => trim(s: line) != "")) } } + ProcessOutputAbsent => ResidueAbsent + other => ResidueUnreadable { reason: "network residue listing failed" as NonEmptyStr } + } +} + +fn observe_network_ruleset() -> RulesetObservation { + match run_shell_command_capture(command: runner_microvm_nft_list_command(), transport: LocalExec) { + ProcessOutputPresent { text: body } => match sha256sum_stdin_digest_via_shell(content: body) { + Sha256FileDigest { digest: hashed } => match sha256_hex_digest(hex: hashed.hex as String) { + Present { value: actual_digest } => RulesetListed { digest: actual_digest } + Absent => RulesetUnreadable { reason: "listed ruleset digest malformed" as NonEmptyStr } + } + other => RulesetUnreadable { reason: "listed ruleset digest unavailable" as NonEmptyStr } + } + other => RulesetUnreadable { reason: "converged nft table could not be listed" as NonEmptyStr } + } +} + +// Actual host reads, joined by the existing subject and converge receipt. A missing tool or +// denied kernel read is unreadable; no exited process or no-carrier state substitutes for these. +fn observe_slot_network_readings(subject: SlotNetworkSanitationSubject) -> SlotNetworkReadings { + let net = runner_slot_network(slot: subject.subject.slot) + let ipv6 = Filesystem.Read(path: runner_slot_ipv6_disabled_proc_path(net: net)) + let ipv6_standing = if !ipv6.success { TapIpv6Unread { reason: "tap IPv6 policy unreadable" as NonEmptyStr } } + else if trim(s: ipv6.content) == "1" { TapIpv6Disabled } + else if trim(s: ipv6.content) == "0" { TapIpv6Enabled } + else { TapIpv6Unread { reason: "tap IPv6 policy malformed" as NonEmptyStr } } + SlotNetworkReadings { + tap: TapReading { subject: subject.subject, observation: observe_tap_holders(tap: net.tap_device) }, + tap_ipv6: TapIpv6Reading { subject: subject.subject, observation: ipv6_standing }, + guest_originated_conntrack: SlotResidueReading { subject: subject.subject, observation: observe_network_residue(command: runner_slot_guest_originated_conntrack_list_command(net: net)) }, + guest_destined_conntrack: SlotResidueReading { subject: subject.subject, observation: observe_network_residue(command: runner_slot_guest_destined_conntrack_list_command(net: net)) }, + guest_neighbour: SlotResidueReading { subject: subject.subject, observation: observe_network_residue(command: runner_slot_guest_neighbour_show_command(net: net)) }, + expectations: HostResidueReading { host: subject.converged.host, generation: subject.converged.generation, observation: observe_network_residue(command: runner_host_expectation_list_command()) }, + ruleset: RulesetReading { host: subject.converged.host, generation: subject.converged.generation, observation: observe_network_ruleset() }, + } +} diff --git a/dag/gunbc/runner/runner_microvm_slot_controller.dag b/dag/gunbc/runner/runner_microvm_slot_controller.dag index 982c75b32f2..53b37148709 100644 --- a/dag/gunbc/runner/runner_microvm_slot_controller.dag +++ b/dag/gunbc/runner/runner_microvm_slot_controller.dag @@ -1,5 +1,7 @@ module gunbc.runner_microvm_slot_controller +import gunbc.runner_microvm_network_receipt_store { read_network_receipt, NetworkReceiptAbsent, NetworkReceiptPresent, StoredNetworkReceiptUnreadable } + import gunbc.workspace_commissioning_executor_fence { workspace_executor_claim_controller, WorkspaceExecutorControllerAdmitted, WorkspaceExecutorControllerRefused } import std.durable_exclusive_hold { DurableHoldOwnerRef } @@ -405,18 +407,12 @@ type NetworkSubjectResolution fn resolve_network_subject(slot: RunnerSlotIdentity) -> NetworkSubjectResolution uses fs: Filesystem { - let read = Filesystem.Read(path: runner_microvm_converged_receipt_path) - if !read.success { - NetworkSubjectRefused { refusal: NetworkReceiptUnreadable { path: runner_microvm_converged_receipt_path, detail: read.error } } - } else { - match parse_converged_slot_network(text: read.content, candidates: fabric_execution_slot_identities()) { - ConvergedSlotNetworkMalformed { cause: c } => - NetworkSubjectRefused { refusal: NetworkReceiptMalformed { path: runner_microvm_converged_receipt_path, cause: c } } - ConvergedSlotNetworkParsed { receipt: r } => - match slot_network_sanitation_subject(slot: slot, receipt: r) { - SanitationSubjectRefused { refusal: rf } => NetworkSubjectRefused { refusal: SanitationSubjectNotAdmitted { refusal: rf } } - SanitationSubjectAdmitted { subject: s } => NetworkSubjectResolved { subject: s } - } + match read_network_receipt(candidates: fabric_execution_slot_identities()) { + NetworkReceiptAbsent => NetworkSubjectRefused { refusal: NetworkReceiptUnreadable { path: runner_microvm_converged_receipt_path, detail: "network receipt is absent" } } + StoredNetworkReceiptUnreadable { reason } => NetworkSubjectRefused { refusal: NetworkReceiptUnreadable { path: runner_microvm_converged_receipt_path, detail: reason } } + NetworkReceiptPresent { receipt } => match slot_network_sanitation_subject(slot: slot, receipt: receipt) { + SanitationSubjectRefused { refusal } => NetworkSubjectRefused { refusal: SanitationSubjectNotAdmitted { refusal: refusal } } + SanitationSubjectAdmitted { subject } => NetworkSubjectResolved { subject: subject } } } } diff --git a/dag/gunbc/workspace_commissioning_dispatch_operations.dag b/dag/gunbc/workspace_commissioning_dispatch_operations.dag index 218b660aae4..1db6d1dda21 100644 --- a/dag/gunbc/workspace_commissioning_dispatch_operations.dag +++ b/dag/gunbc/workspace_commissioning_dispatch_operations.dag @@ -1,7 +1,7 @@ module gunbc.workspace_commissioning_dispatch_operations import std.types { String, NonEmptyStr, List } -import gunbc.executor_privileged_operation { ExecutorPrivilegedOperation, RunGunbcEntryAs, ReadFile, executor_privileged_operation_sudoers_line } +import gunbc.executor_privileged_operation { ExecutorPrivilegedOperation, RunGunbcEntryAs, ReadFileSha256, executor_privileged_operation_sudoers_line } import gunbc.runner_microvm_slot_unit { microvm_controller_gunbc_binary, microvm_controller_owner } import gunbc.fleet_generation_execution { fleet_generation_writer } import gunbc.fleet_posix_accounts { fleet_posix_operator_user } @@ -16,10 +16,10 @@ fn workspace_dispatch_operation(release_dir: NonEmptyStr, function: NonEmptyStr) data workspace_dispatch_grant_path: String = "/etc/sudoers.d/gunbc-workspace-commissioning" data workspace_dispatch_functions: List = ["workspace_dispatch_plan_cli", "workspace_dispatch_stage_cli", "workspace_dispatch_complete_cli", "workspace_dispatch_offer_cli", "workspace_dispatch_allocation_plan_cli", "workspace_dispatch_allocation_readback_cli"] -// The grant file remains root:root 0440. Readers may inspect this exact public policy, never +// The grant file remains root:root 0440. Readers may digest this exact policy, never // another file; the command and its sudoers row are projections of this single operation. fn workspace_dispatch_grant_read_operation() -> ExecutorPrivilegedOperation { - ReadFile { path: workspace_dispatch_grant_path as NonEmptyStr } + ReadFileSha256 { path: workspace_dispatch_grant_path as NonEmptyStr } } fn workspace_dispatch_grant_content(release_dir: NonEmptyStr) -> String { diff --git a/dag/gunbc/workspace_commissioning_executor_input.dag b/dag/gunbc/workspace_commissioning_executor_input.dag index bbe9dd3f110..a7d6ea592bc 100644 --- a/dag/gunbc/workspace_commissioning_executor_input.dag +++ b/dag/gunbc/workspace_commissioning_executor_input.dag @@ -7,7 +7,10 @@ import gunbc.workspace_commissioning_dispatch_operations { workspace_dispatch_gr import gunbc.executor_privileged_operation { executor_privileged_operation_elevated_command } import gunbc.command_runner { run_shell_command_capture } import extdeps.exec.command { LocalExec } -import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } +import extdeps.shell.exec { ProcessOutcome, ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } + +import extdeps.crypto.hash { Digest, sha256sum_line_digest } +import extdeps.tools.sha256sum { sha256sum_stdin_digest_via_shell, Sha256FileDigest, Sha256FileDigestUnavailable } import std.types { String, NonEmptyStr, CommitSha, Bool } import std.algebra { trim } @@ -94,15 +97,26 @@ fn workspace_executor_owned_text_equals(path: String, expected: String) -> Bool } } -// Installer readback runs as the job user. Sudoers custody is checked before using the exact -// read-only grant; changing permissions or substituting a successful install is not a readback. +// Captured command text trims trailing whitespace. Digest the file under its exact grant, +// and the expected bytes through stdin, so a missing or added final newline still refuses. +fn workspace_executor_grant_digest_matches(expected: Digest, observed: ProcessOutcome) -> Bool { + match observed { + ProcessOutputPresent { text } => match sha256sum_line_digest(line: text) { + Present { value: actual } => actual == expected + Absent => false + } + ProcessOutputAbsent => false + ProcessRefused { exit_code: _, stderr: _ } => false + } +} + fn workspace_executor_grant_matches(expected: String) -> Bool { match read_store_ownership(path: workspace_dispatch_grant_path) { - OwnershipRead { ownership: StoreRootOwned } => match run_shell_command_capture( - command: executor_privileged_operation_elevated_command(op: workspace_dispatch_grant_read_operation()), transport: LocalExec) { - ProcessOutputPresent { text } => text == expected - ProcessOutputAbsent => false - ProcessRefused { exit_code: _, stderr: _ } => false + OwnershipRead { ownership: StoreRootOwned } => match sha256sum_stdin_digest_via_shell(content: expected) { + Sha256FileDigest { digest } => workspace_executor_grant_digest_matches(expected: digest, + observed: run_shell_command_capture(command: executor_privileged_operation_elevated_command( + op: workspace_dispatch_grant_read_operation()), transport: LocalExec)) + Sha256FileDigestUnavailable { path: _, reason: _ } => false } OwnershipRead { ownership: StoreNotRootOwned { owner_uid: _, mode: _ } } => false OwnershipUnread { reason: _ } => false diff --git a/dag/gunbc/workspace_initial_sanitation.dag b/dag/gunbc/workspace_initial_sanitation.dag index 8d4876faeea..448daa2280e 100644 --- a/dag/gunbc/workspace_initial_sanitation.dag +++ b/dag/gunbc/workspace_initial_sanitation.dag @@ -20,7 +20,7 @@ import gunbc.runner_microvm_lifecycle { } import gunbc.runner_microvm_lifecycle_realize { observe_cgroup_subtree, mount_point_is_within } import gunbc.runner_microvm_network { SlotNetworkSanitationSubject, SlotNetworkReadback, slot_network_verdict } -import gunbc.runner_microvm_network_observe { observe_slot_network_readings } +import gunbc.runner_microvm_network_sanitation_observe { observe_slot_network_readings } import product.fabric.sanitation { SanitationObservation, MountsRemoved, WritableLayersDeleted, SecretsRemoved } // Observation identity is content-addressed; observed_at separately carries freshness. A clean diff --git a/dag/test/claim/runner/runner_microvm_network_apply_witness_test.dag b/dag/test/claim/runner/runner_microvm_network_apply_witness_test.dag index 50289b2a590..4630235326a 100644 --- a/dag/test/claim/runner/runner_microvm_network_apply_witness_test.dag +++ b/dag/test/claim/runner/runner_microvm_network_apply_witness_test.dag @@ -6,7 +6,7 @@ import gunbc.fleet_host_identity { operator_host_srv1 } import gunbc.runner_microvm_network_converge { runner_microvm_staging_dir } import gunbc.runner_microvm_network_apply { MicrovmNetworkApplyAdmission, MicrovmNetworkApplyAdmitted, MicrovmNetworkApplyRefused, - microvm_network_apply_admission, staging_ancestor_paths, + microvm_network_apply_admission, microvm_network_local_admission, staging_ancestor_paths, path_metadata_ok, OperationResult, ApplyLeg, ApplyLegRan, ApplyLegRefused, operation_result_ok, runner_microvm_root_only_probe_argv, @@ -148,3 +148,17 @@ test fn the_root_only_probe_is_elevated_and_keeps_its_predicate() -> Bool { "-maxdepth", "0", "-type", "f", "-user", "root", "-group", "root", "-not", "-perm", "/022", ] } + + +test fn local_network_apply_requires_root_on_the_selected_host() -> Bool { + admitted(a: microvm_network_local_admission(host: "srv1", observed_host: "srv1", effective_uid: "0", + expected_revision: a_revision, checked_out_revision: a_revision)) + && !admitted(a: microvm_network_local_admission(host: "srv1", observed_host: "srv2", effective_uid: "0", + expected_revision: a_revision, checked_out_revision: a_revision)) + && !admitted(a: microvm_network_local_admission(host: "srv1", observed_host: "srv1", effective_uid: "999", + expected_revision: a_revision, checked_out_revision: a_revision)) + && !admitted(a: microvm_network_local_admission(host: "srv1", observed_host: "srv1", effective_uid: "", + expected_revision: a_revision, checked_out_revision: a_revision)) + && !admitted(a: microvm_network_local_admission(host: "srv1", observed_host: "srv1", effective_uid: "0", + expected_revision: a_revision, checked_out_revision: another_revision)) +} diff --git a/dag/test/claim/runner/runner_microvm_network_converge_witness_test.dag b/dag/test/claim/runner/runner_microvm_network_converge_witness_test.dag index b090ea28346..fd144807697 100644 --- a/dag/test/claim/runner/runner_microvm_network_converge_witness_test.dag +++ b/dag/test/claim/runner/runner_microvm_network_converge_witness_test.dag @@ -383,3 +383,14 @@ test fn a_probe_that_did_not_answer_refuses_under_its_own_name() -> Bool { && outcome_label(o: converge_outcome(slots: [unread_route], h: clean_host())) == "route-unread" && outcome_label(o: converge_outcome(slots: [clean_slot(i: 1)], h: host_with(fwd: IpForwardOff, digest: Present { value: converged_digest }, auto: kernel_absent()))) == "no-forwarding" } + + +// networkd must not re-enable IPv6 after sysctl has disabled it on the TAP. +test fn staged_tap_policy_disables_networkd_ipv6_autoconfiguration() -> Bool { + let files = converge_staged_files(slots: [slot(i: 13)], host: runner_microvm_shared_host_grain, tap_owner: "ghrunner") + any(files, file => file.path == "/var/lib/gunbc-microvm-network-stage/gunbc-tap13.network" + && string_contains(s: file.content, pattern: "LinkLocalAddressing=no\n") + && string_contains(s: file.content, pattern: "IPv6AcceptRA=no\n")) + && any(files, file => file.path == "/var/lib/gunbc-microvm-network-stage/gunbc-tap13-ipv6.conf" + && string_contains(s: file.content, pattern: "disable_ipv6 = 1")) +} diff --git a/dag/test/claim/workspace_dispatch_fleet_witness_test.dag b/dag/test/claim/workspace_dispatch_fleet_witness_test.dag index e5117f79142..1943edf9944 100644 --- a/dag/test/claim/workspace_dispatch_fleet_witness_test.dag +++ b/dag/test/claim/workspace_dispatch_fleet_witness_test.dag @@ -2,6 +2,10 @@ module test.claim.workspace_dispatch_fleet_witness import gunbc.fleet_converge_scope { WorkspaceSlotCommissioningOnly } +import extdeps.crypto.hash { sha256_digest } +import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } +import gunbc.workspace_commissioning_executor_input { workspace_executor_grant_digest_matches } + import std.types { Bool, String } import gunbc.fleet_converge_plan { ScopeDecoded, @@ -53,9 +57,22 @@ test fn installation_readback_grants_only_the_exact_policy_file() -> Bool { let op = workspace_dispatch_grant_read_operation() let argv = executor_privileged_operation_argv(op: op) let grant = executor_privileged_operation_sudoers_line(grantee: fleet_generation_writer(), op: op) as String - argv == ["/usr/bin/cat", "/etc/sudoers.d/gunbc-workspace-commissioning"] + argv == ["/usr/bin/sha256sum", "--", "/etc/sudoers.d/gunbc-workspace-commissioning"] && string_contains(s: dispatch_grant_fixture(), pattern: grant) - && ends_with(s: grant, suffix: "/usr/bin/cat /etc/sudoers.d/gunbc-workspace-commissioning") + && ends_with(s: grant, suffix: "/usr/bin/sha256sum -- /etc/sudoers.d/gunbc-workspace-commissioning") && !string_contains(s: grant, pattern: "*") && !string_contains(s: grant, pattern: "state-writer-mac-key") } + + +// Independent SHA-256 fixtures for "policy\n" and "policy": captured text normalizes their +// trailing newline, but the file digest must still distinguish them. +test fn policy_digest_readback_preserves_final_newline_identity() -> Bool { + let expected = sha256_digest(hex: "c82fc52c78bf8154d4dd7d8766c422ab151052d72098d72ded237aafcd78e4e0") + workspace_executor_grant_digest_matches(expected: expected, + observed: ProcessOutputPresent { text: "c82fc52c78bf8154d4dd7d8766c422ab151052d72098d72ded237aafcd78e4e0 /etc/sudoers.d/gunbc-workspace-commissioning" }) + && !workspace_executor_grant_digest_matches(expected: expected, + observed: ProcessOutputPresent { text: "823412d1eacb67956220e532959f0104603057c88704863ca38e7cd188fda812 /etc/sudoers.d/gunbc-workspace-commissioning" }) + && !workspace_executor_grant_digest_matches(expected: expected, observed: ProcessOutputAbsent) + && !workspace_executor_grant_digest_matches(expected: expected, observed: ProcessRefused { exit_code: 1, stderr: "denied" }) +} diff --git a/dag/test/claim/workspace_network_producer_witness_test.dag b/dag/test/claim/workspace_network_producer_witness_test.dag index 59551d312d6..7269b53b761 100644 --- a/dag/test/claim/workspace_network_producer_witness_test.dag +++ b/dag/test/claim/workspace_network_producer_witness_test.dag @@ -1,7 +1,11 @@ module test.claim.workspace_network_producer_witness_test import std.types { Bool } -import gunbc.runner_microvm_network_observe { tap_holders_reading } +import gunbc.runner_microvm_network_observe { network_read_outcome } +import gunbc.host_effect { SshShell } +import gunbc.executor_privileged_operation { ReadNftRuleset } +import extdeps.shell.exec { ProcessRefused } +import gunbc.runner_microvm_network_sanitation_observe { tap_holders_reading } import gunbc.runner_microvm_network { TapPresentUnheld, TapHeld, TapHoldersUnreadable } import extdeps.linux.tun_fdinfo { TunDescriptorCount, TunDescriptorUnreadable, TunCensusMultiQueue } @@ -10,3 +14,11 @@ test fn network_producer_requires_successful_tun_census() -> Bool { && (match tap_holders_reading(census: TunDescriptorCount { count: 1, exited: 0 }) { TapHeld { holders: n } => n == 1 other => false }) && (match tap_holders_reading(census: TunDescriptorUnreadable { cause: TunCensusMultiQueue }) { TapHoldersUnreadable { reason: _ } => true other => false }) } + + +test fn unanchored_network_observation_refuses_before_opening_a_session() -> Bool { + match network_read_outcome(transport: SshShell { ssh_host: "unexpected-host" }, op: ReadNftRuleset) { + ProcessRefused { exit_code: _, stderr: _ } => true + other => false + } +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index dd5b35c4b5e..203bc100504 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -18,6 +18,8 @@ collected that refusal; its green workflow conclusion is not a boot receipt. | Workflow dispatch | Main declares 28 inputs; GitHub accepts at most 25 | Keep printer selection separate and carry its five operation fields in one `printer_request` JSON input | | Guest image | Builder exists; no image delivery in controller install | Build under fakeroot, archive kernel/rootfs/digest receipt, pin the measured rootfs, verify installed bytes and custody | | Job selection | Controller install queues the shared credentialed job as well as its dedicated job | Derive shared job selection from the existing mode-to-job authority | +| Network receipt | The historical TAP/firewall installation exists, but the controller receipt is absent and the workflow observer entry was overwritten | Restore the observer, publish a root-owned initial receipt from live reads, and keep later sanitation observation separate | +| TAP IPv6 | Installed sysctl disables IPv6; networkd leaves it enabled on the live TAP | Disable networkd IPv6 autoconfiguration in the same generated TAP policy, then converge and read back | | Controller | Installed revision `cfb9ff80652fe0a74093f08cade64f59d9695266` | Install this PR through the existing controller installer | | Slot budget | `fabric-cell-srv1-13.slice` has no installed unit file | Converge the modeled execution-cell boundary; a synthesized systemd slice is not proof | | Cell inventory | Read-only plan `local-2026-10-09T03:18:35Z` refuses foreign `fabric-cell-zp444.slice` | Establish ownership before retiring it; the empty transient slice has no unit file or remaining journal, and is not silently ignored | @@ -43,7 +45,17 @@ empty commissioning prestate. Do not restore the old CI reservation service: not authorize them. The installer checks the reviewed kernel and rootfs pins after copying into the root-owned release directory, then independently reads back the image and installed executor. -3. Converge the execution-cell boundary and authenticated state service, then run +3. Converge the execution-cell boundary and authenticated state service. On srv1, + the local administrator can run `runner_microvm_network_plan_local_wet`, + review `target/microvm-network-local-plan.txt`, then run + `runner_microvm_network_apply_local_wet` from + `dag/gunbc/runner/runner_microvm_network_apply.dag`. Both require `host=srv1`, + `expected_revision` equal to the clean checkout's HEAD, and effective UID 0. + Run `runner_microvm_network_initialize_local_wet` from + `dag/gunbc/runner/runner_microvm_network_observe.dag` with those same arguments + to publish the controller receipt from live readback. This uses the installed + root-owned receipt directory and refuses replacement of a different generation. + These local entries require no SSH agent. Then run `workspace_commissioning_plan` and its reviewed `apply` on srv1. The existing operator-local fleet plan/apply path is available for pre-merge validation; do not widen the main-only GCP login to run branch workflows. @@ -228,3 +240,38 @@ and the existing prohibition on workflow-created operator intent. The verified `a85cc481af5` compiler binary can interpret this DAG-only repair; its build identity remains `a85cc481af5`, separately from the new installed source revision. It is not represented as a rebuilt compiler pack for the new tree. + + +The read-only commissioning preflight at `a85cc481af5` refused because +`/var/lib/gunbc/microvm-network/converged-slot-network.txt` and its parent directory +are absent. The network workflow still names `runner_microvm_network_observe_wet`, +but that function was overwritten when the same module was repurposed for +per-attempt sanitation. The previous implementation was recovered from source +revision `34ac2fefab51e912e9499dd23aa92bb3ebdac824`; sanitation keeps its own current +producer. The repair restores the workflow entry and adds an explicit local root +initializer which observes the host, publishes once with create-only semantics, +and verifies existing generations against fresh observations without overwriting. + +The live `gunbc-tap13` holds `172.30.13.1/30`, and the existing microVM nft service +is active. However, `disable_ipv6` reads `0` despite the installed sysctl file +requesting `1`. systemd 255's [link IPv6 decision](https://github.com/systemd/systemd/blob/v255/src/network/networkd-link.c) +and [sysctl writer](https://github.com/systemd/systemd/blob/v255/src/network/networkd-sysctl.c) +show why the generated `.network` file must also disable link-local IPv6: +networkd re-enables IPv6 when its configuration calls for it. The local network +plan/apply entries use the existing staged-file and ordered-operation producers, +require root on the named host at the named revision, and use no forwarded agent. +Network changes have not yet been applied. + +The `edad20b0ded` controller installer completed its mutations and installed the +exact policy read grant. A real `ghrunner` invocation can now read that policy, +and `visudo` accepts it, but exact adapter readback still refused. The next +boundary diagnosis identified `bounded_shell_host_drain`'s `trim_end` capture contract: +the final newline cannot survive a `cat` through that transport. Policy readback +now compares the actual file's SHA-256 with a digest of the expected bytes. The +exact grant becomes `sha256sum -- `; it grants neither arbitrary +file reads nor the protected-state credential. The focused validation passes all +34 checks across network apply/convergence, observer production, policy readback +and allocation dispatch. The final network-path retry passes its 12 checks after +using the existing command executor and effective-UID reader. All 8,272 source +files parse, and formatting checks pass. Live reinstallation and network +convergence are next; no VM has been commissioned, allocated or booted. From 4f98ce8a7c4eb82a2f932a88a5a453d11b272ee5 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 07:04:49 +0000 Subject: [PATCH 11/19] Wait for controller exit before commissioning bind readback --- ...ontroller_in_its_single_process_budget.dag | 15 +++++++++++ .../workspace_commissioning_executor.dag | 16 ++++++++++-- docs/plans/srv1-workspace-vm-bringup.md | 26 +++++++++++++++++++ 3 files changed, 55 insertions(+), 2 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/bind_helper_overlaps_the_controller_in_its_single_process_budget.dag diff --git a/dag/gunbc/recurring_failure_mode/bind_helper_overlaps_the_controller_in_its_single_process_budget.dag b/dag/gunbc/recurring_failure_mode/bind_helper_overlaps_the_controller_in_its_single_process_budget.dag new file mode 100644 index 00000000000..51b94c7d20f --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/bind_helper_overlaps_the_controller_in_its_single_process_budget.dag @@ -0,0 +1,15 @@ +module gunbc.recurring_failure_mode.bind_helper_overlaps_the_controller_in_its_single_process_budget + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data bind_helper_overlaps_the_controller_in_its_single_process_budget: RecurringFailureMode = RecurringFailureMode { + identity: "bind_helper_overlaps_the_controller_in_its_single_process_budget" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607, pre-commissioning review: workspace_executor_submit_once started the controller and immediately invoked the bind interpreter in the same 24 GiB controller slice. The slice budget derives from one compiler preparation plus headroom; scheduling a second preparation does not establish that their aggregate fits. Bringup interpreters were using multiple GiB each. No live commissioning OOM is claimed: the defect was found before submission.", + "The controller already publishes its invocation through workspace_executor_claim_controller before commissioning effects. The executor's bind remains useful for independent readback and a controller that fails before claiming, but it need not overlap controller execution. Move it after the existing bounded wait has observed the exact invocation inactive, with MainPID zero and no pending job. The bind helper still independently checks manager identity, executor membership and exact mutation generation, so replacement after the wait refuses. The prior unbound-submission fence/drain recovery path remains intact.", + "Executed discriminator: the local bringup experiment target/srv1-bringup-evidence/executor-bind-order-controls.py runs the actual emitted Bash against bounded helper and manager doubles. The predecessor attempts bind while active and exits 73; the revised start and drain paths complete. Replaced invocation and surviving PID refuse before finish; a readable but incorrect finish result refuses. Wait, settlement and completion replay retain their behavior. These are manual native-transport controls, not hermetic floor evidence or a measurement of recovery-path memory.", + "Rung: the normal start path now orders the bind process after observed controller termination; live systemd state remains an external readback obligation. This does not establish aggregate resource qualification for every recovery helper or externally started controller. Ceiling and next trigger: a consumed resource admission model for concurrently resident controller/helper processes, discriminated on all normal and recovery routes. Source-level sequencing alone cannot prove a host-memory bound over external starts.", + ], + evidence: [], +} diff --git a/dag/gunbc/workspace_commissioning_executor.dag b/dag/gunbc/workspace_commissioning_executor.dag index 8d538bab8bf..67ede96538e 100644 --- a/dag/gunbc/workspace_commissioning_executor.dag +++ b/dag/gunbc/workspace_commissioning_executor.dag @@ -12,7 +12,10 @@ import v2.workflow.orchestration_bash_emit_support { orch_bash_do, orch_bash_emi // settle/complete are permitted only from recorded post-readback obligations; they skip the // old controller wait because the slot may already have been legitimately reused. The helper // must verify the durable release/generation receipts, not reconstruct them from live guest state. -// bind persists the actual slot invocation. finish owns strong readback, credential removal, +// The controller claims its invocation in the journal before any commissioning effect. After +// it exits, bind confirms that invocation (or records a controller that failed before claiming). +// This ordering keeps the controller and bind interpreter out of the same bounded slice at once. +// finish owns strong readback, credential removal, // mutation settlement and generation publication (including recovery after publication). // None of these helpers is called in the waiting loop. The systemd-owned executor retains the // existing fleet flock even when the workflow disconnects. Failure NEVER releases the durable @@ -88,7 +91,6 @@ data workspace_executor_submit_once: String = join([ " executor_observe || exit 66", " [[ $executor_seen =~ ^[a-f0-9]\{32}$ && $executor_seen != \"$executor_expected\" ]] || exit 67", " executor_expected=$executor_seen", - " timeout --kill-after=5s \"$\{executor_polls}s\" \"$executor_helper\" bind \"$INVOCATION_ID\" \"$executor_expected\"", "fi", ], "\n") @@ -120,6 +122,15 @@ data workspace_executor_settle: String = join([ "printf '%s\\n' \"$executor_terminal\"", ], "\n") +// The bounded wait already proved this exact invocation quiescent. bind independently rereads +// systemd and the held mutation, so replacement between the wait and this helper still refuses. +// A disconnected submitter retains the same unbound-submission fence/drain recovery obligation. +data workspace_executor_bind_completed_invocation: String = join([ + "if [[ $executor_action == start ]]; then", + " timeout --kill-after=5s \"$\{executor_polls}s\" \"$executor_helper\" bind \"$INVOCATION_ID\" \"$executor_expected\"", + "fi", +], "\n") + // The interpreter emits the transport once when building the reviewed artifact, not while // waiting on a host. Foreign command leaves contain transport mechanics only; no readiness, // reservation, provenance or fleet-generation policy is implemented in these Bash commands. @@ -131,6 +142,7 @@ fn workspace_commissioning_executor_script() -> String { orch_bash_do(command: workspace_executor_drain_submission), orch_bash_do(command: workspace_executor_submit_once), orch_bash_do(command: workspace_executor_await_invocation), + orch_bash_do(command: workspace_executor_bind_completed_invocation), orch_bash_do(command: workspace_executor_settle), ], on_failure: FailFast, diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 203bc100504..6ba593bc3f8 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -275,3 +275,29 @@ and allocation dispatch. The final network-path retry passes its 12 checks after using the existing command executor and effective-UID reader. All 8,272 source files parse, and formatting checks pass. Live reinstallation and network convergence are next; no VM has been commissioned, allocated or booted. + +Controller installation at `682c32644e84da04dd4e73d334413057a659dbf3` now exits +successfully, including exact executor/policy and guest-image readback. The network +plan at that revision also succeeds. Comparison with the installed files finds +only `LinkLocalAddressing=no` and `IPv6AcceptRA=no` added to the ten enrolled TAP +configurations; firewall, TAP addresses and IPv4 forwarding remain identical. +The reviewed apply is running; network convergence is not yet claimed. + +Pre-commissioning review found the native executor started its bind interpreter +alongside the controller in the same 24 GiB slice. The controller already binds +its journal identity before commissioning effects. The executor now waits for +that invocation to terminate before its independent bind/readback, keeping those +two preparations sequential without changing the resource ceiling or recovery +fences. This fixes the normal start overlap; recovery-path aggregate memory has +not been measured. No live commissioning OOM is claimed. + +The manual transport experiment used `export_executor` from +`dag/test/claim/workspace_commissioning_executor_witness_test.dag` and the local +`target/srv1-bringup-evidence/executor-bind-order-controls.py` driver against +`target/workspace-commissioning-executor.sh`. The Python driver is local dev +tooling, excluded from the committed substrate by `gunbc.repo_workspace`; these +results are not claimed as an ongoing CI regression gate. +All eight revised-script cases pass: normal start, invocation replacement, +surviving PID, refused finish, wait, settle, complete and drain. The optional +`--reject-active-bind-script` control against the installed `682c32644e8` +predecessor refuses its premature bind, distinguishing the two schedules. From 83dcbb4d6ab51e0e05a519dd930dc41ef0539eaa Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 07:22:00 +0000 Subject: [PATCH 12/19] fix(workspace): replace prior local network staging atomically --- ...y_writer_used_for_reconcilable_staging.dag | 14 ++++ .../runner/runner_microvm_network_apply.dag | 57 +++++++++++---- ...unner_microvm_network_local_stage_test.dag | 70 +++++++++++++++++++ docs/plans/srv1-workspace-vm-bringup.md | 16 ++++- 4 files changed, 143 insertions(+), 14 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/create_only_writer_used_for_reconcilable_staging.dag create mode 100644 dag/test/manual/runner_microvm_network_local_stage_test.dag diff --git a/dag/gunbc/recurring_failure_mode/create_only_writer_used_for_reconcilable_staging.dag b/dag/gunbc/recurring_failure_mode/create_only_writer_used_for_reconcilable_staging.dag new file mode 100644 index 00000000000..fef29767b38 --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/create_only_writer_used_for_reconcilable_staging.dag @@ -0,0 +1,14 @@ +module gunbc.recurring_failure_mode.create_only_writer_used_for_reconcilable_staging + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data create_only_writer_used_for_reconcilable_staging: RecurringFailureMode = RecurringFailureMode { + identity: "create_only_writer_used_for_reconcilable_staging" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607: the local network apply at 682c32644e8 refused File exists for its prior root-owned staging artifacts. No network install operation ran. Filesystem.WriteOwnerOnly creates an exclusive 0600 file; the caller chose it for its custody contract but also assumed replacement semantics it does not provide. A fresh staging directory would hide the defect until the next converge.", + "Repair at the write realization: mktemp creates a unique 0600 inode inside the already-verified staging directory, Filesystem.Write fills that inode, and the existing mv_exact_destination_command atomically publishes it on the same filesystem. Scope admission applies to the destination and temporary path. Failures remove only this invocation's temporary inode and report cleanup refusal; final exact byte readback and the outer root-custody readback remain required.", + "Discriminator: test.manual.runner_microvm_network_local_stage_test.local_stage_replaces_owned_bytes_wet executes the real production writer for creation, replacement with different bytes and replay, checks 0600 mode and trailing newlines, refuses a destination outside the supplied scope, and checks no temporary inode remains. This is a manual native filesystem control, not a hermetic CI claim. Rung: external byte/custody facts are observed and refused; constructor-level distinction between exclusive creation and replacement remains the ceiling for callers choosing a filesystem operation. Trigger: operation admission requires the caller's intended publication semantics rather than relying on the writer's name.", + ], + evidence: [], +} diff --git a/dag/gunbc/runner/runner_microvm_network_apply.dag b/dag/gunbc/runner/runner_microvm_network_apply.dag index b11325d6c66..122b91956bd 100644 --- a/dag/gunbc/runner/runner_microvm_network_apply.dag +++ b/dag/gunbc/runner/runner_microvm_network_apply.dag @@ -13,6 +13,8 @@ import std.content_hash { Sha256Digest, sha256_hex_digest } import extdeps.filesystem.filesystem_io { Filesystem } import extdeps.tools.sha256sum { Sha256FileDigest, Sha256FileDigestUnavailable, sha256sum_stdin_digest_via_shell } import extdeps.git +import extdeps.shell +import extdeps.tools.gnu_coreutils { mv_exact_destination_command } import extdeps.git.inspect import product.host_identity { HostIdentity } import gunbc.output_policy { ExpectSuccess } @@ -438,25 +440,54 @@ fn write_staged_file( TypedRemoteFileConvergenceRefused { step: st, reason: why } => StagedWriteRefused { step: st, reason: why } TypedRemoteFileConverged { converged: _ } => StagedWritten } - LocalShell => match typed_remote_file_write_seal(path: file.path as FilePath, - content: file.content, allowed_scope: runner_microvm_admin_staging_scope) { - TypedRemoteFileWriteAdmissionRefused { cause } => StagedWriteRefused { step: "admission", reason: cause as String } - TypedRemoteFileWriteAdmitted { write: _ } => { - let written = Filesystem.WriteOwnerOnly(path: file.path as String, content: file.content) - if !written.success { StagedWriteRefused { step: "local root stage", reason: written.error } } - else { - let read = Filesystem.Read(path: file.path as String) - if read.success && read.content == file.content { StagedWritten } - else { StagedWriteRefused { step: "local root readback", reason: "staged bytes differ or cannot be read" } } - } - } - } + LocalShell => write_local_staged_file(file: file, scope: runner_microvm_admin_staging_scope) SshShell { ssh_host: _ } => StagedWriteRefused { step: "transport", reason: "unanchored SSH is not an administrator transport" } EmitArtifactThenThinRun { bootstrap: _, invocation: _ } => StagedWriteRefused { step: "transport", reason: "artifact-only transport cannot write host files" } }, } } +// mktemp creates the private 0600 inode; ordinary Write fills that already-owned inode. +// Rename publishes it atomically on the same filesystem, including on a later converge. +// WriteOwnerOnly is create-exclusive and therefore cannot update a prior staging artifact. +// Production establishes root custody of the scope and every ancestor before calling this. +fn write_local_staged_file(file: StagedFile, scope: FileTreeScope) -> StagedWriteOutcome { + match typed_remote_file_write_seal(path: file.path as FilePath, content: file.content, allowed_scope: scope) { + TypedRemoteFileWriteAdmissionRefused { cause } => StagedWriteRefused { step: "admission", reason: cause as String } + TypedRemoteFileWriteAdmitted { write: _ } => { + let temporary = shell.Mktemp.FileInDirectory(dir: scope.root_path as FilePath) + if !temporary.success { StagedWriteRefused { step: "local stage", reason: "private staging inode could not be created" } } + else { + let path = trim(s: temporary.path as String) + match typed_remote_file_write_seal(path: path as FilePath, content: file.content, allowed_scope: scope) { + TypedRemoteFileWriteAdmissionRefused { cause } => StagedWriteRefused { step: "local temporary admission", reason: cause as String } + TypedRemoteFileWriteAdmitted { write: _ } => { + let written = Filesystem.Write(path: path, content: file.content) + if !written.success { local_stage_cleanup_refusal(path: path, reason: written.error) } + else { match run_shell_command_observe_outcome( + command: mv_exact_destination_command(source: path, destination: file.path as String), transport: LocalExec) { + ProcessNotSpawned { program: _, cause } => local_stage_cleanup_refusal(path: path, reason: cause as String) + ProcessRan { observation } => if observation.exit_code != 0 { + local_stage_cleanup_refusal(path: path, reason: observation.stderr) + } else { + let read = Filesystem.Read(path: file.path as String) + if read.success && read.content == file.content { StagedWritten } + else { StagedWriteRefused { step: "local readback", reason: "staged bytes differ or cannot be read" } } + } + } } + } + } + } + } + } +} + +fn local_stage_cleanup_refusal(path: String, reason: String) -> StagedWriteOutcome { + let removed = shell.Remove.FileForce(path: path as FilePath) + StagedWriteRefused { step: "local stage", reason: if removed.success { reason } + else { concat(reason, "; private staging inode cleanup also refused") } } +} + // ── The readback ───────────────────────────────────────────────────────────────────────────────── type RulesetReadback diff --git a/dag/test/manual/runner_microvm_network_local_stage_test.dag b/dag/test/manual/runner_microvm_network_local_stage_test.dag new file mode 100644 index 00000000000..d9f46a196d2 --- /dev/null +++ b/dag/test/manual/runner_microvm_network_local_stage_test.dag @@ -0,0 +1,70 @@ +module test.manual.runner_microvm_network_local_stage_test + +import std.types { String, NonEmptyStr, FilePath, Bool } +import std.algebra { trim } +import std.process { ProcessExit, ExitSuccess, exit_failure } +import extdeps.shell +import extdeps.filesystem.filesystem_io { Filesystem, filesystem_listing_names_entry } +import extdeps.tools.stat { stat_mode_command } +import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } +import gunbc.command_runner { run_shell_command_capture, LocalExec } +import gunbc.file_access { FileTreeScope } +import gunbc.runner_microvm_network_converge { StagedFile } +import gunbc.runner_microvm_network_apply { write_local_staged_file, StagedWriteOutcome, StagedWritten, StagedWriteRefused } + +fn stage_succeeded(result: StagedWriteOutcome) -> Bool { + match result { StagedWritten => true StagedWriteRefused { step: _, reason: _ } => false } +} + +fn owner_only(path: String) -> Bool { + match run_shell_command_capture(command: stat_mode_command(path: path), transport: LocalExec) { + ProcessOutputPresent { text } => trim(s: text) == "600" + ProcessOutputAbsent => false + ProcessRefused { exit_code: _, stderr: _ } => false + } +} + +// The admitted directory exists, so removing scope admission would actually write the +// out-of-scope destination. A missing directory would refuse for an unrelated reason. +fn outside_scope_refuses(root: String) -> Bool { + let allowed = shell.Mktemp.DirWithTemplate(template: concat(root, "/allowed.XXXXXX") as FilePath) + if !allowed.success { false } + else { + let scope = FileTreeScope { root_path: trim(s: allowed.path as String) as NonEmptyStr } + let refused = write_local_staged_file(file: StagedFile { path: concat(root, "/outside") as NonEmptyStr, content: "no" }, scope: scope) + let children = Filesystem.List(path: scope.root_path as String) + let parent = Filesystem.List(path: root) + !stage_succeeded(result: refused) && children.success && trim(s: children.entries) == "" + && parent.success && !filesystem_listing_names_entry(listing: parent.entries, name: "outside") + } +} + +// Real local filesystem effects in one freshly created scratch directory. No root, network, +// sudo or host configuration is touched. This CLI control is not a hermetic floor witness. +fn local_stage_replaces_owned_bytes_wet() -> ProcessExit { + let temporary = shell.Mktemp.Dir() + if !temporary.success { exit_failure(reason: "local staging control could not create its scratch directory") } + else { + let root = trim(s: temporary.path as String) + let path = concat(root, "/staged") + let scope = FileTreeScope { root_path: root as NonEmptyStr } + let first = write_local_staged_file(file: StagedFile { path: path as NonEmptyStr, content: "prior\n" }, scope: scope) + let first_read = Filesystem.Read(path: path) + let first_mode = owner_only(path: path) + let next = StagedFile { path: path as NonEmptyStr, content: "replacement\n\n" } + let second = write_local_staged_file(file: next, scope: scope) + let second_read = Filesystem.Read(path: path) + let second_mode = owner_only(path: path) + let replay = write_local_staged_file(file: next, scope: scope) + let replay_read = Filesystem.Read(path: path) + let listing = Filesystem.List(path: root) + let scope_refused = outside_scope_refuses(root: root) + let removed = shell.Remove.RecursiveForce(path: root as FilePath) + if stage_succeeded(result: first) && first_read.success && first_read.content == "prior\n" && first_mode + && stage_succeeded(result: second) && second_read.success && second_read.content == next.content && second_mode + && stage_succeeded(result: replay) && replay_read.success && replay_read.content == next.content + && scope_refused && listing.success && trim(s: listing.entries) == "staged" && removed.success { + ExitSuccess + } else { exit_failure(reason: "local staging failed create, replacement, replay, custody, scope refusal or cleanup control") } + } +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 6ba593bc3f8..cf490d4111e 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -281,7 +281,9 @@ successfully, including exact executor/policy and guest-image readback. The netw plan at that revision also succeeds. Comparison with the installed files finds only `LinkLocalAddressing=no` and `IPv6AcceptRA=no` added to the ten enrolled TAP configurations; firewall, TAP addresses and IPv4 forwarding remain identical. -The reviewed apply is running; network convergence is not yet claimed. +The reviewed apply refused before any network installation: its staging writer +used exclusive creation against existing root-owned staging files. Network +convergence is not yet claimed. Pre-commissioning review found the native executor started its bind interpreter alongside the controller in the same 24 GiB slice. The controller already binds @@ -301,3 +303,15 @@ All eight revised-script cases pass: normal start, invocation replacement, surviving PID, refused finish, wait, settle, complete and drain. The optional `--reject-active-bind-script` control against the installed `682c32644e8` predecessor refuses its premature bind, distinguishing the two schedules. + +The local network staging writer now creates a private temporary inode in the +verified directory, fills it, and atomically replaces the exact staged destination. +The existing scope admission, root-custody checks and exact byte readback remain +required. The manual native filesystem control +`test.manual.runner_microvm_network_local_stage_test.local_stage_replaces_owned_bytes_wet` +passes creation, changed-content replacement, replay, 0600 mode, trailing-newline +preservation, out-of-scope refusal and scratch cleanup. The out-of-scope control +uses an existing allowed directory, so it distinguishes admission refusal from a +missing-directory failure. It runs as an ordinary user in a fresh scratch directory; +this is manual evidence, not an enrolled CI claim. The live network apply must +still pass with the repaired writer before commissioning. From f99e440ce59874b10d0c9c65c27a3217a0a1cc3e Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 08:09:05 +0000 Subject: [PATCH 13/19] fix(workspace): decode network receipts with the host population --- ...pt_decoded_against_workload_population.dag | 14 +++++++ .../runner/runner_microvm_network_observe.dag | 4 +- .../runner_microvm_network_receipt_store.dag | 32 ++++++++++----- .../runner/runner_microvm_slot_controller.dag | 2 +- .../workspace_commissioning_dispatch.dag | 2 +- dag/gunbc/workspace_commissioning_fleet.dag | 10 ++--- ...etwork_receipt_population_witness_test.dag | 39 +++++++++++++++++++ dag/test/fixture/srv1_network_receipt.dag | 6 +++ docs/plans/srv1-workspace-vm-bringup.md | 25 ++++++++++++ 9 files changed, 116 insertions(+), 18 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/network_receipt_decoded_against_workload_population.dag create mode 100644 dag/test/claim/workspace_network_receipt_population_witness_test.dag create mode 100644 dag/test/fixture/srv1_network_receipt.dag diff --git a/dag/gunbc/recurring_failure_mode/network_receipt_decoded_against_workload_population.dag b/dag/gunbc/recurring_failure_mode/network_receipt_decoded_against_workload_population.dag new file mode 100644 index 00000000000..994409f4ab8 --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/network_receipt_decoded_against_workload_population.dag @@ -0,0 +1,14 @@ +module gunbc.recurring_failure_mode.network_receipt_decoded_against_workload_population + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data network_receipt_decoded_against_workload_population: RecurringFailureMode = RecurringFailureMode { + identity: "network_receipt_decoded_against_workload_population" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607: live network convergence at 83dcbb4d6ab published ten enrolled srv1 TAPs (1 through 9 and 13). Commissioning refused immediately after reading the receipt: resolve_network_subject supplied fabric_execution_slot_identities, whose srv1 member is only 13. The receipt parsed successfully for its publisher and failed for its consumer before sanitation. Network enrollment and workload admission had been conflated at decoding.", + "Repair: the root-authoritative receipt reader takes a host, derives candidates through the same runner_slot_identities_of deployment authority as the publisher, and checks the decoded receipt host. A caller can no longer substitute its workload roster. Fabric execution membership remains separately enforced by the slot controller. Initial publication reads back through this same host reader; it does not replace the existing generation.", + "Discriminator: workspace_network_receipt_population_witness_test feeds the captured live ten-slot receipt to the production host decoder and selects the workspace sanitation subject; the old fabric-only candidate list refuses those same bytes. Separate controls refuse another receipt host and an unenrolled slot. Rung: the arbitrary caller-population choice is removed from the store interface; external receipt bytes still require observed custody and decoding. Ceiling: structural separation of network enrollment from workload admission. Trigger: every future network receipt consumer uses the host reader instead of supplying workload membership.", + ], + evidence: [], +} diff --git a/dag/gunbc/runner/runner_microvm_network_observe.dag b/dag/gunbc/runner/runner_microvm_network_observe.dag index 283cf0b2c57..521f2ed76fa 100644 --- a/dag/gunbc/runner/runner_microvm_network_observe.dag +++ b/dag/gunbc/runner/runner_microvm_network_observe.dag @@ -186,7 +186,7 @@ fn network_read_outcome(transport: HostEffectTransport, op: ExecutorPrivilegedOp fn runner_microvm_network_initialize_local_wet(host: String, expected_revision: String) -> ProcessExit { match microvm_network_local_subject(host: host, expected_revision: expected_revision) { MicrovmNetworkApplyRefused { cause } => exit_failure(reason: cause) - MicrovmNetworkApplyAdmitted { subject } => match read_network_receipt(candidates: subject.slots) { + MicrovmNetworkApplyAdmitted { subject } => match read_network_receipt(host: subject.host) { StoredNetworkReceiptUnreadable { reason } => exit_failure(reason: reason) NetworkReceiptPresent { receipt } => initialize_local_network_receipt(subject: subject, generation: receipt.generation as NonEmptyStr) NetworkReceiptAbsent => match clock_now_probed_at() { @@ -206,7 +206,7 @@ fn initialize_local_network_receipt(subject: MicrovmNetworkApplySubject, generat if !artifact.success { exit_failure(reason: artifact.error) } else { match outcome { ConvergeRefused { refusal: _ } => exit_failure(reason: receipt_text(o: outcome)) - ConvergedNetwork { receipt } => match publish_initial_network_receipt(receipt: receipt, candidates: subject.slots) { + ConvergedNetwork { receipt } => match publish_initial_network_receipt(receipt: receipt) { NetworkReceiptPresent { receipt: _ } => ExitSuccess NetworkReceiptAbsent => exit_failure(reason: "network receipt was not published") StoredNetworkReceiptUnreadable { reason } => exit_failure(reason: reason) diff --git a/dag/gunbc/runner/runner_microvm_network_receipt_store.dag b/dag/gunbc/runner/runner_microvm_network_receipt_store.dag index 597ae85af81..bdec7762a3f 100644 --- a/dag/gunbc/runner/runner_microvm_network_receipt_store.dag +++ b/dag/gunbc/runner/runner_microvm_network_receipt_store.dag @@ -1,8 +1,10 @@ module gunbc.runner_microvm_network_receipt_store -import std.types { String, NonEmptyStr, Bool, List } +import std.types { String, NonEmptyStr, Bool } import std.decl_ref { decl_ref } -import gunbc.build_cache_instance { RunnerSlotIdentity } +import product.host_identity { HostIdentity } +import gunbc.runner_host_deploy { runner_host_deploy_for, runner_slot_identities_of, + RunnerHostDeployFound, RunnerHostDeployUnmodeled, RunnerHostDeployRefused } import gunbc.file_access { path_ancestor_prefixes } import gunbc.runner_microvm_cell_readiness { StoreRootOwned, StoreNotRootOwned } import gunbc.runner_microvm_cell_readiness_store { read_store_ownership, OwnershipRead, OwnershipUnread, entry_is_root_authoritative } @@ -32,7 +34,22 @@ fn network_receipt_ancestors_owned() -> Bool { }) } -fn read_network_receipt(candidates: List) -> NetworkReceiptRead { +// The receipt covers the host's enrolled network population, including runner TAPs that +// are not fabric execution cells. Consumers name the host, never a narrower workload roster. +fn parse_host_network_receipt(text: String, host: HostIdentity) -> NetworkReceiptRead { + match runner_host_deploy_for(host: host) { + RunnerHostDeployUnmodeled { host: _ } => StoredNetworkReceiptUnreadable { reason: "network receipt host has no enrolled deployment" } + RunnerHostDeployRefused { host: _, cause } => StoredNetworkReceiptUnreadable { reason: cause } + RunnerHostDeployFound { deploy } => match parse_converged_slot_network(text: text, + candidates: runner_slot_identities_of(deploy: deploy)) { + ConvergedSlotNetworkParsed { receipt } => if receipt.host == host { NetworkReceiptPresent { receipt: receipt } } + else { StoredNetworkReceiptUnreadable { reason: "network receipt names another host" } } + ConvergedSlotNetworkMalformed { cause } => StoredNetworkReceiptUnreadable { reason: to_string(cause) } + } + } +} + +fn read_network_receipt(host: HostIdentity) -> NetworkReceiptRead { if !network_receipt_ancestors_owned() { StoredNetworkReceiptUnreadable { reason: "network receipt ancestors are absent or outside root custody" } } else { let read = Filesystem.Read(path: runner_microvm_converged_receipt_path) @@ -43,10 +60,7 @@ fn read_network_receipt(candidates: List) -> NetworkReceiptR FilesystemExactPathKindUnrecognized { path: _, observed: _, error } => StoredNetworkReceiptUnreadable { reason: error } FilesystemExactPathRead { path: _, content } => match entry_is_root_authoritative(path: runner_microvm_converged_receipt_path) { Present { value: reason } => StoredNetworkReceiptUnreadable { reason: reason as String } - Absent => match parse_converged_slot_network(text: content, candidates: candidates) { - ConvergedSlotNetworkParsed { receipt } => NetworkReceiptPresent { receipt: receipt } - ConvergedSlotNetworkMalformed { cause } => StoredNetworkReceiptUnreadable { reason: to_string(cause) } - } + Absent => parse_host_network_receipt(text: content, host: host) } } } @@ -54,7 +68,7 @@ fn read_network_receipt(candidates: List) -> NetworkReceiptR // Initial publication is atomic and never replaces a generation a controller may already use. // A rerun verifies the stored generation against fresh host readings before reaching this entry. -fn publish_initial_network_receipt(receipt: ConvergedSlotNetwork, candidates: List) -> NetworkReceiptRead +fn publish_initial_network_receipt(receipt: ConvergedSlotNetwork) -> NetworkReceiptRead admit_callers: [decl_ref(module_path: "gunbc.runner_microvm_network_observe", decl_name: "initialize_local_network_receipt")] { if !network_receipt_ancestors_owned() { StoredNetworkReceiptUnreadable { reason: "network receipt store is not installed under root custody" } } @@ -69,7 +83,7 @@ fn publish_initial_network_receipt(receipt: ConvergedSlotNetwork, candidates: Li FilesystemCreateKindUnrecognized { path: _, observed: _, error: _ } => false } if !published { StoredNetworkReceiptUnreadable { reason: "network receipt publication refused" } } - else { match read_network_receipt(candidates: candidates) { + else { match read_network_receipt(host: receipt.host) { NetworkReceiptPresent { receipt: actual } => if render_converged_slot_network(c: actual) == text { NetworkReceiptPresent { receipt: actual } } else { StoredNetworkReceiptUnreadable { reason: "existing network generation differs; explicit recovery is required" } } diff --git a/dag/gunbc/runner/runner_microvm_slot_controller.dag b/dag/gunbc/runner/runner_microvm_slot_controller.dag index 53b37148709..0af2f88c179 100644 --- a/dag/gunbc/runner/runner_microvm_slot_controller.dag +++ b/dag/gunbc/runner/runner_microvm_slot_controller.dag @@ -407,7 +407,7 @@ type NetworkSubjectResolution fn resolve_network_subject(slot: RunnerSlotIdentity) -> NetworkSubjectResolution uses fs: Filesystem { - match read_network_receipt(candidates: fabric_execution_slot_identities()) { + match read_network_receipt(host: slot.host) { NetworkReceiptAbsent => NetworkSubjectRefused { refusal: NetworkReceiptUnreadable { path: runner_microvm_converged_receipt_path, detail: "network receipt is absent" } } StoredNetworkReceiptUnreadable { reason } => NetworkSubjectRefused { refusal: NetworkReceiptUnreadable { path: runner_microvm_converged_receipt_path, detail: reason } } NetworkReceiptPresent { receipt } => match slot_network_sanitation_subject(slot: slot, receipt: receipt) { diff --git a/dag/gunbc/workspace_commissioning_dispatch.dag b/dag/gunbc/workspace_commissioning_dispatch.dag index 116a5922ff1..34abb8e9359 100644 --- a/dag/gunbc/workspace_commissioning_dispatch.dag +++ b/dag/gunbc/workspace_commissioning_dispatch.dag @@ -180,7 +180,7 @@ fn workspace_dispatch_plan_cli() -> CliWireResponse { workspace_dispatch_refused(reason: "plan revision is malformed") } else { match workspace_commissioning_candidate(host: host as NonEmptyStr, revision: revision as CommitSha) { WorkspaceCommissioningFleetObserved { basis } => CliWirePrintable { bytes: workspace_commissioning_basis_rows(basis: basis), exit: ExitSuccess } - WorkspaceCommissioningFleetRefused { reason: _ } => workspace_dispatch_refused(reason: "privileged commissioning observation refused") + WorkspaceCommissioningFleetRefused { reason } => workspace_dispatch_refused(reason: concat("privileged commissioning observation refused: ", reason)) } } } } diff --git a/dag/gunbc/workspace_commissioning_fleet.dag b/dag/gunbc/workspace_commissioning_fleet.dag index 14839d08417..fb09ea3b730 100644 --- a/dag/gunbc/workspace_commissioning_fleet.dag +++ b/dag/gunbc/workspace_commissioning_fleet.dag @@ -12,7 +12,7 @@ import gunbc.build_cache_instance { RunnerSlotIdentity, runner_slot_instance_nam import gunbc.runner_slot_allocation { workspace_execution_slot_identities, gunbc_runner_slot_desired_for, gunbc_microvm_shakedown_slot_memory_max } import gunbc.runner_slot_desired { CpuQuotaResolved, SlotCpuQuotaUnbounded } import gunbc.runner_microvm_host_ready { MicrovmHostBound, MicrovmHostBindingRefused } -import gunbc.runner_microvm_slot_controller { bind_slot_host, resolve_network_subject, NetworkSubjectResolved, NetworkSubjectRefused } +import gunbc.runner_microvm_slot_controller { bind_slot_host, resolve_network_subject, NetworkSubjectResolved, NetworkSubjectRefused, refusal_text } import gunbc.runner_microvm_slot_unit { microvm_controller_root, microvm_controller_gunbc_binary, microvm_slot_unit_instance_name } import gunbc.live_deploy.release_locus { approval_broker_release_dir, approval_broker_tree_receipt_path_for, RevisionBoundAtEmission } import gunbc.auth.approval_broker_cutover { observe_broker_tree_receipt, TreeReceiptObserved, TreeReceiptMalformed, TreeReceiptUnreadable } @@ -106,9 +106,9 @@ fn workspace_commissioning_candidate_slot(slot: RunnerSlotIdentity, revision: Co TreeReceiptMalformed { detail: _ } => WorkspaceCommissioningFleetRefused { reason: "installed tree receipt is unreadable" } } } - WorkspaceInitialSanitationUnavailable { reason: _ } => WorkspaceCommissioningFleetRefused { reason: "slot sanitation is unreadable" } + WorkspaceInitialSanitationUnavailable { reason } => WorkspaceCommissioningFleetRefused { reason: concat("slot sanitation is unreadable: ", reason) } } - NetworkSubjectRefused { refusal: _ } => WorkspaceCommissioningFleetRefused { reason: "slot network subject is unavailable" } + NetworkSubjectRefused { refusal } => WorkspaceCommissioningFleetRefused { reason: concat("slot network subject is unavailable: ", refusal_text(r: refusal)) } } MicrovmHostBindingRefused { cause: _ } => WorkspaceCommissioningFleetRefused { reason: "commissioning candidate is not on its host" } } @@ -188,9 +188,9 @@ fn workspace_commissioning_basis_observe(provenance: WorkspaceCommissioningProve } WorkspaceCommissioningStartUnreadable { reason: _ } => WorkspaceCommissioningFleetRefused { reason: "controller prestate is unreadable" } } } - WorkspaceInitialSanitationUnavailable { reason: _ } => WorkspaceCommissioningFleetRefused { reason: "fresh sanitation is unreadable" } + WorkspaceInitialSanitationUnavailable { reason } => WorkspaceCommissioningFleetRefused { reason: concat("fresh sanitation is unreadable: ", reason) } } - NetworkSubjectRefused { refusal: _ } => WorkspaceCommissioningFleetRefused { reason: "network subject is unreadable" } + NetworkSubjectRefused { refusal } => WorkspaceCommissioningFleetRefused { reason: concat("network subject is unreadable: ", refusal_text(r: refusal)) } } } } WorkspaceExecutorJournalUnreadable { reason: _ } => WorkspaceCommissioningFleetRefused { reason: "executor journal is unreadable" } diff --git a/dag/test/claim/workspace_network_receipt_population_witness_test.dag b/dag/test/claim/workspace_network_receipt_population_witness_test.dag new file mode 100644 index 00000000000..43791d83f86 --- /dev/null +++ b/dag/test/claim/workspace_network_receipt_population_witness_test.dag @@ -0,0 +1,39 @@ +module test.claim.workspace_network_receipt_population_witness_test + +import std.types { Bool, String } +import gunbc.fleet_host_identity { operator_host_srv1 } +import gunbc.runner_slot_allocation { fabric_execution_slot_identities, srv1_microvm_shakedown_slot } +import gunbc.runner_microvm_network_receipt_store { parse_host_network_receipt, NetworkReceiptPresent, StoredNetworkReceiptUnreadable } +import gunbc.runner_microvm_network { parse_converged_slot_network, ConvergedSlotNetworkMalformed, + slot_network_sanitation_subject, SanitationSubjectAdmitted } +import test.fixture.srv1_network_receipt { observed_srv1_network_receipt } + +// Actual publisher bytes discriminate the previous consumer's narrower fabric-cell roster. +// Parsing the host receipt does not authorize another slot for fabric execution. +test fn published_host_network_population_reaches_the_workspace_consumer() -> Bool { + (match parse_converged_slot_network(text: observed_srv1_network_receipt, + candidates: fabric_execution_slot_identities()) { ConvergedSlotNetworkMalformed { cause: _ } => true other => false }) + && (match parse_host_network_receipt(text: observed_srv1_network_receipt, host: operator_host_srv1) { + NetworkReceiptPresent { receipt } => receipt.slots.count() == 10 + && (match slot_network_sanitation_subject(slot: srv1_microvm_shakedown_slot(), receipt: receipt) { + SanitationSubjectAdmitted { subject: _ } => true other => false + }) + other => false + }) +} + +test fn host_network_reader_refuses_another_receipt_host() -> Bool { + let text = join(split(s: observed_srv1_network_receipt, delimiter: "host=srv1"), "host=srv2") + match parse_host_network_receipt(text: text, host: operator_host_srv1) { + StoredNetworkReceiptUnreadable { reason } => reason == "network receipt names another host" + other => false + } +} + +test fn host_network_reader_refuses_an_unenrolled_network_slot() -> Bool { + let text = join(split(s: observed_srv1_network_receipt, delimiter: "slot=srv1-13"), "slot=srv1-99") + match parse_host_network_receipt(text: text, host: operator_host_srv1) { + StoredNetworkReceiptUnreadable { reason: _ } => true + other => false + } +} diff --git a/dag/test/fixture/srv1_network_receipt.dag b/dag/test/fixture/srv1_network_receipt.dag new file mode 100644 index 00000000000..2416e9c32c0 --- /dev/null +++ b/dag/test/fixture/srv1_network_receipt.dag @@ -0,0 +1,6 @@ +module test.fixture.srv1_network_receipt + +import std.types { String } + +// Captured root-owned srv1 receipt, 2026-10-09T07:45:19Z. Fixture bytes are not live authority. +data observed_srv1_network_receipt: String = "gunbc.converged_slot_network.v1\nhost=srv1\ngeneration=local-83dcbb4d6ab51e0e05a519dd930dc41ef0539eaa-2026-10-09T07:45:19Z\nnft_digest=da7a00c54f267b39227a02557ebef28c90282376673cef9c78784caca7479099\nhelper_auto=absent_from_kernel:6.8.0-138-generic\nhelper_nft=counted:0\nhelper_xtables=counted:0\nhelper_userspace=counted:0\nhelper_expectations=counted:0\nslot=srv1-01|tap=gunbc-tap1|guest=172.30.1.2|mac=02:00:00:00:00:01|host_cidr=172.30.1.1/30|route=172.30.1.0/30 proto kernel scope link src 172.30.1.1 linkdown|ipv6=disabled\nslot=srv1-02|tap=gunbc-tap2|guest=172.30.2.2|mac=02:00:00:00:00:02|host_cidr=172.30.2.1/30|route=172.30.2.0/30 proto kernel scope link src 172.30.2.1 linkdown|ipv6=disabled\nslot=srv1-03|tap=gunbc-tap3|guest=172.30.3.2|mac=02:00:00:00:00:03|host_cidr=172.30.3.1/30|route=172.30.3.0/30 proto kernel scope link src 172.30.3.1 linkdown|ipv6=disabled\nslot=srv1-04|tap=gunbc-tap4|guest=172.30.4.2|mac=02:00:00:00:00:04|host_cidr=172.30.4.1/30|route=172.30.4.0/30 proto kernel scope link src 172.30.4.1 linkdown|ipv6=disabled\nslot=srv1-05|tap=gunbc-tap5|guest=172.30.5.2|mac=02:00:00:00:00:05|host_cidr=172.30.5.1/30|route=172.30.5.0/30 proto kernel scope link src 172.30.5.1 linkdown|ipv6=disabled\nslot=srv1-06|tap=gunbc-tap6|guest=172.30.6.2|mac=02:00:00:00:00:06|host_cidr=172.30.6.1/30|route=172.30.6.0/30 proto kernel scope link src 172.30.6.1 linkdown|ipv6=disabled\nslot=srv1-07|tap=gunbc-tap7|guest=172.30.7.2|mac=02:00:00:00:00:07|host_cidr=172.30.7.1/30|route=172.30.7.0/30 proto kernel scope link src 172.30.7.1 linkdown|ipv6=disabled\nslot=srv1-08|tap=gunbc-tap8|guest=172.30.8.2|mac=02:00:00:00:00:08|host_cidr=172.30.8.1/30|route=172.30.8.0/30 proto kernel scope link src 172.30.8.1 linkdown|ipv6=disabled\nslot=srv1-09|tap=gunbc-tap9|guest=172.30.9.2|mac=02:00:00:00:00:09|host_cidr=172.30.9.1/30|route=172.30.9.0/30 proto kernel scope link src 172.30.9.1 linkdown|ipv6=disabled\nslot=srv1-13|tap=gunbc-tap13|guest=172.30.13.2|mac=02:00:00:00:00:0d|host_cidr=172.30.13.1/30|route=172.30.13.0/30 proto kernel scope link src 172.30.13.1 linkdown|ipv6=disabled\n" diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index cf490d4111e..839e0c3e955 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -315,3 +315,28 @@ uses an existing allowed directory, so it distinguishes admission refusal from a missing-directory failure. It runs as an ordinary user in a fresh scratch directory; this is manual evidence, not an enrolled CI claim. The live network apply must still pass with the repaired writer before commissioning. + +Controller installation at `83dcbb4d6ab51e0e05a519dd930dc41ef0539eaa` passes +its complete readback. The fresh network plan has identical staged bytes and +operation argv to the reviewed `682c32644e8` plan. Network apply now passes, +including replacement of existing staging files; its live nft digest is +`da7a00c54f267b39227a02557ebef28c90282376673cef9c78784caca7479099`. +Independent readback observes `gunbc-tap13.disable_ipv6=1`, IPv4 forwarding enabled, +and both `gunbc-microvm-nft.service` and fabric storage active with successful +results. The initializer passes and publishes root:root `0644` +`/var/lib/gunbc/microvm-network/converged-slot-network.txt`, generation +`local-83dcbb4d6ab51e0e05a519dd930dc41ef0539eaa-2026-10-09T07:45:19Z`. +Its live readings cover all ten enrolled TAPs with IPv6 disabled and count zero +helper attachments and expectations across all observed surfaces. The initial +commissioning plan refused before writing an artifact; no commissioning effect or +allocation has started. Its coordinator compiled in 451 seconds and its installed +privileged observer in 327 seconds. The observer stopped immediately after reading +the network receipt. The controller supplied `fabric_execution_slot_identities()` +to the parser, which admits only slot 13 on srv1; the publisher's enrolled network +population also contains TAPs 1 through 9. The reader now takes the host and derives +the network population from the same deployment authority as the publisher. +Workload admission remains a separate controller check. The regression fixture is +the actual published ten-slot receipt, including the predecessor's rejecting parse. +All three focused checks pass: the complete host population reaches the workspace +consumer, another receipt host refuses, and an unenrolled network slot refuses. +Commissioning diagnostics now retain the underlying network/sanitation refusal. From ce5bf60b5072a5038d0c4d9d3f0451932444e6b4 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 08:52:29 +0000 Subject: [PATCH 14/19] fix(workspace): preserve and repair the enrolled cell CPU quota --- dag/gunbc/fabric/fabric_cell_converge.dag | 8 ++- dag/gunbc/fabric/fabric_cell_effect.dag | 63 +++++++++++++----- dag/gunbc/fleet/fleet_converge_plan.dag | 2 +- ...cell_add_overrides_the_slot_cpu_policy.dag | 14 ++++ .../fabric_cell_effect_witness_test.dag | 66 ++++++++++++++++--- .../fleet_converge_plan_witness_test.dag | 4 +- docs/plans/srv1-workspace-vm-bringup.md | 20 ++++++ 7 files changed, 146 insertions(+), 31 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/cell_add_overrides_the_slot_cpu_policy.dag diff --git a/dag/gunbc/fabric/fabric_cell_converge.dag b/dag/gunbc/fabric/fabric_cell_converge.dag index f1bb9f67170..b8a3d526805 100644 --- a/dag/gunbc/fabric/fabric_cell_converge.dag +++ b/dag/gunbc/fabric/fabric_cell_converge.dag @@ -209,6 +209,12 @@ fn fabric_cell_member_ownership(member: FabricCellMember) -> Optional // every other slot, and a digest rendered from the fleet row alone would report the converged // shakedown cell as drifted on every observation (or, worse, converge it to the wrong ceiling). fn fabric_cell_resource_boundary_state_digest(slot: RunnerSlotIdentity) -> NonEmptyStr { + fabric_cell_resource_boundary_state_digest_with_quota(slot: slot, quota: gunbc_runner_slot_desired_for(slot: slot).cpu_quota) +} + +// Also identifies the precise predecessor admitted by the CPU-only repair. All other +// resource fields retain this slot's desired values; no unrelated drift is normalized. +fn fabric_cell_resource_boundary_state_digest_with_quota(slot: RunnerSlotIdentity, quota: RunnerSlotCpuQuota) -> NonEmptyStr { let desired = gunbc_runner_slot_desired_for(slot: slot) fabric_cell_boundary_state_digest_of( memory_max: to_string(byte_size_count(b: desired.memory_max)), @@ -216,7 +222,7 @@ fn fabric_cell_resource_boundary_state_digest(slot: RunnerSlotIdentity) -> NonEm memory_swap_max: to_string(byte_size_count(b: desired.memory_swap_max)), tasks_max: to_string(desired.tasks_max), cpu_weight: to_string(desired.cpu_weight), - cpu_quota_per_sec: fabric_cell_desired_cpu_quota_identity(quota: desired.cpu_quota), + cpu_quota_per_sec: fabric_cell_desired_cpu_quota_identity(quota: quota), ) } diff --git a/dag/gunbc/fabric/fabric_cell_effect.dag b/dag/gunbc/fabric/fabric_cell_effect.dag index 68b0aed7b51..d145fed2eff 100644 --- a/dag/gunbc/fabric/fabric_cell_effect.dag +++ b/dag/gunbc/fabric/fabric_cell_effect.dag @@ -13,7 +13,8 @@ import extdeps.systemd.unit_file { slice_cpu_quota_unbounded, } import gunbc.build_cache_instance { RunnerSlotIdentity } -import gunbc.runner_slot_allocation { gunbc_runner_slot_desired_for } +import gunbc.runner_slot_allocation { gunbc_runner_slot_desired_for, identity_is_workspace_execution_member } +import gunbc.change_realization { address_continuity, classify_change, HostResourceSubject, ChangeRealized, InPlaceUpdate } import gunbc.runner_slot_desired { gunbc_runner_slot_desired, RunnerSlotCpuQuota, CpuQuotaResolved, SlotCpuQuotaUnbounded } import gunbc.membership_reconcile { MemberAction, @@ -35,6 +36,8 @@ import gunbc.fabric_cell_converge { fabric_cell_attempt_root_path, fabric_cell_slice_name, fabric_cell_resource_address_wire, + fabric_cell_member_key, fabric_cell_desired_state_digest, + fabric_cell_boundary_address_digest_of, fabric_cell_resource_boundary_state_digest_with_quota, } data fabric_cell_effect_authority_disposition: ConstructionMechanism = SingleAuthority @@ -95,7 +98,7 @@ fn fabric_cell_base_directory_effect() -> FabricCellEffect { // DESIGN 4b forbids. // // FIVE BASE DIRECTIVES PLUS THE CPU QUOTA LINE, ALWAYS SIX. Every magnitude comes from -// gunbc_runner_slot_desired -- the same row fabric_cell_resource_boundary_state_digest renders +// gunbc_runner_slot_desired_for -- the same row fabric_cell_resource_boundary_state_digest renders // into the desired member digest -- so the directives this effect applies and the digest reconcile // compares are two projections of one authority and cannot disagree by drift. // @@ -110,9 +113,8 @@ fn fabric_cell_base_directory_effect() -> FabricCellEffect { // arms nothing can produce. // // WHAT THIS DOES NOT DO, STATED SO THE RESET IS NOT READ AS A REPAIR (review F-R1, 2026-09-19). -// The six directives are rendered only for MemberAdded. An EXISTING boundary that differs from the -// desired row only by a finite CPU quota reconciles as MemberChanged, and every MemberChanged at -// this address is refused as CellChangeNotYetRealizable by fabric_cell_address_realization below -- +// The six directives are rendered for MemberAdded. An EXISTING boundary that differs from the +// desired unbounded row by a finite CPU quota reconciles as MemberChanged and is still refused -- // so a previously installed finite quota is DETECTED (the digest compares "infinity" against the // host's CPUQuotaPerSecUSec identity and reports drift) and is NOT cleared by this module. That is // the inherited change-realization boundary, not a new one: gunbc.change_realization has not @@ -124,6 +126,9 @@ fn fabric_cell_base_directory_effect() -> FabricCellEffect { // reported srv4 runner-unit census found unlimited quotas; the fabric boundary // (fabric-cell-srv3-06.slice) requires its own readback, and this row does not establish // fleet-wide quota conformity -- a cell found carrying a finite quota sits behind this block. +// The separately admitted reverse direction repairs an enrolled workspace cell from unlimited +// CPU to its declared finite quota, with every other resource field already exact. Its runtime +// address is preserved, classified as InPlaceUpdate, and only the quota directive is emitted. // // WHY DIRECTIVES AND NOT A SystemdUnitFile: the effect's production realization is // `systemctl set-property =...` -- systemd renders and owns the persisted @@ -196,7 +201,8 @@ type FabricCellAddressRealization // decides WHETHER there is one. Neither axis is collapsed into the other, so a new address and a // new action each fail to compile here rather than inheriting an arm that happened to fit. // -// TEARDOWN AND CHANGE REFUSE RATHER THAN BEING REALIZED. ADDITION always realizes: directories +// TEARDOWN AND GENERAL CHANGE REFUSE. The CPU-only narrowing below is the sole admitted change. +// ADDITION always realizes: directories // by path, the slice by the six directives of fabric_cell_slice_desired_directives_for. // Teardown of a cell root is a DESTRUCTIVE effect on a directory that may hold live // attempt state, and change realization is gunbc.change_realization's question -- it classifies a @@ -225,17 +231,9 @@ fn fabric_cell_address_realization( ), } FabricCellResourceBoundaryAddress => - fabric_cell_boundary_add_realization( - slot: m.slot, - quota: gunbc_runner_slot_desired().cpu_quota, - ) - } - MemberChanged { at: _, from: _, to: m } => - AddressRefused { - refusal: CellChangeNotYetRealizable { - address_wire: fabric_cell_resource_address_wire(address: m.address), - }, + fabric_cell_boundary_add_realization(slot: m.slot) } + MemberChanged { at, from, to } => fabric_cell_workspace_quota_repair(at: at, from: from, to: to) MemberRemoved { at: _, from: m } => AddressRefused { refusal: CellTeardownNotYetRealizable { @@ -260,16 +258,45 @@ fn fabric_cell_address_realization( fn fabric_cell_boundary_add_realization( slot: RunnerSlotIdentity, - quota: RunnerSlotCpuQuota, ) -> FabricCellAddressRealization { AddressRealizable { effect: FabricCellSliceEnsure { slice_unit: fabric_cell_slice_name(slot: slot) as NonEmptyStr, - directives: fabric_cell_slice_desired_directives_for_slot(slot: slot, quota: quota), + directives: fabric_cell_slice_desired_directives_of_slot(slot: slot), }, } } +// A finite CPU quota limits scheduling rate; it neither deletes storage nor lowers memory. +// Fleet apply reobserves the exact baseline under its mutation lock. This recognizes only the +// erroneous unlimited predecessor with all other limits already exact, on the same enrolled +// workspace address. It does not admit quota removal, other resource drift or replacement. +fn fabric_cell_workspace_quota_repair(at: String, from: FabricCellMember, to: FabricCellMember) -> FabricCellAddressRealization { + let refused = AddressRefused { refusal: CellChangeNotYetRealizable { + address_wire: fabric_cell_resource_address_wire(address: to.address), + } } + if from.address != FabricCellResourceBoundaryAddress || to.address != FabricCellResourceBoundaryAddress + || from.slot != to.slot || at != fabric_cell_member_key(member: from) || at != fabric_cell_member_key(member: to) + || !identity_is_workspace_execution_member(identity: to.slot) + || to.state_digest != fabric_cell_desired_state_digest(slot: to.slot, address: to.address) + || from.state_digest != fabric_cell_boundary_address_digest_of(slice_name: fabric_cell_slice_name(slot: to.slot), + state: fabric_cell_resource_boundary_state_digest_with_quota(slot: to.slot, quota: SlotCpuQuotaUnbounded)) { + refused + } else { match gunbc_runner_slot_desired_for(slot: to.slot).cpu_quota { + SlotCpuQuotaUnbounded => refused + CpuQuotaResolved { threads } => match classify_change(subject: HostResourceSubject, + subject_label: fabric_cell_slice_name(slot: to.slot) as NonEmptyStr, + continuity: address_continuity(from: from, to: to, address_eq: (a, b) => fabric_cell_member_key(member: a) == fabric_cell_member_key(member: b)), + declared_intermediates: none) { + ChangeRealized { realization: InPlaceUpdate } => AddressRealizable { effect: FabricCellSliceEnsure { + slice_unit: fabric_cell_slice_name(slot: to.slot) as NonEmptyStr, + directives: [slice_cpu_quota(threads: threads)], + } } + other => refused + } + } } +} + // THE ALL-OR-NOTHING FOLD, AND IT IS THE POINT OF THE MODULE RATHER THAN A GUARD ON IT. // // AN EARLIER READING WOULD HAVE EMITTED THE TWO DIRECTORY CREATES AND REFUSED ONLY THE BOUNDARY, diff --git a/dag/gunbc/fleet/fleet_converge_plan.dag b/dag/gunbc/fleet/fleet_converge_plan.dag index afe22136e9d..d75a9ee354b 100644 --- a/dag/gunbc/fleet/fleet_converge_plan.dag +++ b/dag/gunbc/fleet/fleet_converge_plan.dag @@ -1136,7 +1136,7 @@ fn fabric_cell_realization_refusal_line(refusal: FabricCellRealizationRefusal) - CellRosterDuplicateKey { address_wire: w } => concat("fabric-cell ", concat(w, " roster duplicate key")) CellChangeNotYetRealizable { address_wire: w } => - concat("fabric-cell ", concat(w, " change realization undeclared: no staged intermediates (gunbc.change_realization)")) + concat("fabric-cell ", concat(w, " change has no admitted resource transition (gunbc.change_realization)")) CellTeardownNotYetRealizable { address_wire: w } => concat("fabric-cell ", concat(w, " teardown unmodeled: destructive effect over possibly-live attempt state")) } diff --git a/dag/gunbc/recurring_failure_mode/cell_add_overrides_the_slot_cpu_policy.dag b/dag/gunbc/recurring_failure_mode/cell_add_overrides_the_slot_cpu_policy.dag new file mode 100644 index 00000000000..a2c59562fdf --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/cell_add_overrides_the_slot_cpu_policy.dag @@ -0,0 +1,14 @@ +module gunbc.recurring_failure_mode.cell_add_overrides_the_slot_cpu_policy + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data cell_add_overrides_the_slot_cpu_policy: RecurringFailureMode = RecurringFailureMode { + identity: "cell_add_overrides_the_slot_cpu_policy" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607: commissioning at f99e440ce59 refused the cell-boundary digest. Independent systemd readback found CPUQuotaPerSecUSec=infinity while all five other fields matched. The persisted quota drop-in dates to the initial 03:47:23 cell apply, whose FullyApplied terminal recorded successful commands, not full boundary convergence. The MemberAdded realization passed the fleet-wide unlimited quota into a slot-specific renderer, overriding srv1-13's declared four-CPU limit.", + "Repair: the production boundary-add constructor now accepts only a slot and derives every directive from its desired row. The fixture quota renderer remains available for its bounded/unbounded controls. A regression drives the actual MemberAdded dispatch and requires CPUQuota=400%, with no unlimited assignment.", + "Existing-state repair is limited to an enrolled workspace boundary with all non-CPU fields exact and an unlimited predecessor. The same runtime address is classified through gunbc.change_realization as InPlaceUpdate; realization emits only its policy-derived CPUQuota directive. Memory drift, a finite predecessor, quota removal, a moved address and an unreviewed target refuse. It never deletes or recreates the slice. The fleet apply's locked baseline reobservation still gates execution; independent live boundary readback is required before commissioning. Rung: the add constructor no longer accepts a conflicting quota authority, and the bounded repair has discriminating claim controls. External systemd state remains a readback obligation. Ceiling/trigger: complete applied-state readback at the fabric-cell apply terminal, so command success alone cannot be confused with convergence.", + ], + evidence: [], +} diff --git a/dag/test/claim/fabric/fabric_cell_effect_witness_test.dag b/dag/test/claim/fabric/fabric_cell_effect_witness_test.dag index 2253801b435..4d76a15a342 100644 --- a/dag/test/claim/fabric/fabric_cell_effect_witness_test.dag +++ b/dag/test/claim/fabric/fabric_cell_effect_witness_test.dag @@ -3,6 +3,7 @@ module test.claim.fabric_cell_effect_witness import std.types { Bool, String, NonEmptyStr, Int, List, list_length } import gunbc.fleet_host_identity { operator_host_srv3 } import gunbc.build_cache_instance { RunnerSlotIdentity } +import gunbc.runner_slot_allocation { srv1_microvm_shakedown_slot } import gunbc.membership_reconcile { MemberAction, MemberAdded, MemberChanged, MemberRemoved, MemberRemovalRefused, MemberNotOwned, @@ -39,7 +40,8 @@ import gunbc.fabric_cell_effect { fabric_cell_realization, fabric_cell_realization_refusal_count, fabric_cell_plan_effects, - fabric_cell_boundary_add_realization, + fabric_cell_slice_desired_directives_for_slot, + fabric_cell_address_realization, AddressRefused, AddressRealizable, } @@ -273,13 +275,59 @@ fn has_line(lines: List, line: String) -> Bool { } fn boundary_add_directive_lines(quota: RunnerSlotCpuQuota) -> List { - match fabric_cell_boundary_add_realization(slot: srv3_fabric_slot(), quota: quota) { - AddressRefused { refusal: _ } => [] as List - AddressRealizable { effect: e } => - match e { - FabricCellSliceEnsure { slice_unit: _, directives: ds } => - map(ds, d => systemd_slice_directive_line(directive: d)) - FabricCellDirectoryEnsure { path: _, owner: _ } => [] as List - } + map(fabric_cell_slice_desired_directives_for_slot(slot: srv3_fabric_slot(), quota: quota), + d => systemd_slice_directive_line(directive: d)) +} + +fn workspace_boundary_member(digest: NonEmptyStr) -> FabricCellMember { + FabricCellMember { slot: srv1_microvm_shakedown_slot(), address: FabricCellResourceBoundaryAddress, state_digest: digest } +} + +fn workspace_desired_boundary() -> FabricCellMember { + workspace_boundary_member(digest: fabric_cell_desired_state_digest(slot: srv1_microvm_shakedown_slot(), address: FabricCellResourceBoundaryAddress)) +} + +// Independent live read, 2026-10-09: the add path persisted all five other limits but unlimited CPU. +data workspace_unlimited_boundary: NonEmptyStr = "fabric-cell-srv1-13.slice;memory_max=30064771072;memory_high=30064771072;memory_swap_max=0;tasks_max=16384;cpu_weight=100;cpu_quota_per_sec=infinity" + +test fn workspace_boundary_add_uses_its_own_cpu_quota() -> Bool { + let member = workspace_desired_boundary() + match fabric_cell_address_realization(action: MemberAdded { at: fabric_cell_member_key(member: member), to: member }) { + AddressRealizable { effect: FabricCellSliceEnsure { slice_unit, directives } } => slice_unit == "fabric-cell-srv1-13.slice" + && directives.count() == 6 + && has_line(lines: map(directives, d => systemd_slice_directive_line(directive: d)), line: "CPUQuota=400%") + && !has_line(lines: map(directives, d => systemd_slice_directive_line(directive: d)), line: "CPUQuota=") + other => false + } +} + +test fn workspace_unlimited_predecessor_repairs_only_the_cpu_directive() -> Bool { + let target = workspace_desired_boundary() + let change = MemberChanged { at: fabric_cell_member_key(member: target), + from: workspace_boundary_member(digest: workspace_unlimited_boundary), to: target } + match fabric_cell_address_realization(action: change) { + AddressRealizable { effect: FabricCellSliceEnsure { slice_unit, directives } } => slice_unit == "fabric-cell-srv1-13.slice" + && map(directives, d => systemd_slice_directive_line(directive: d)) == ["CPUQuota=400%"] + other => false } } + +test fn workspace_quota_repair_refuses_other_drift_or_an_unreviewed_target() -> Bool { + let target = workspace_desired_boundary() + let old = workspace_boundary_member(digest: workspace_unlimited_boundary) + let key = fabric_cell_member_key(member: target) + let wrong_memory = workspace_boundary_member(digest: join(split(s: workspace_unlimited_boundary as String, + delimiter: "memory_max=30064771072"), "memory_max=4294967296") as NonEmptyStr) + let finite_prior = workspace_boundary_member(digest: join(split(s: workspace_unlimited_boundary as String, + delimiter: "cpu_quota_per_sec=infinity"), "cpu_quota_per_sec=2000000") as NonEmptyStr) + all([ + MemberChanged { at: key, from: wrong_memory, to: target }, + MemberChanged { at: key, from: finite_prior, to: target }, + MemberChanged { at: key, from: target, to: old }, + MemberChanged { at: "another-address", from: old, to: target }, + MemberChanged { at: key, from: old, to: member_at(address: FabricCellResourceBoundaryAddress) }, + ], action => match fabric_cell_address_realization(action: action) { + AddressRefused { refusal: CellChangeNotYetRealizable { address_wire: _ } } => true + other => false + }) +} diff --git a/dag/test/claim/fleet/fleet_converge_plan_witness_test.dag b/dag/test/claim/fleet/fleet_converge_plan_witness_test.dag index 0baa56630a6..a8ee76ddd94 100644 --- a/dag/test/claim/fleet/fleet_converge_plan_witness_test.dag +++ b/dag/test/claim/fleet/fleet_converge_plan_witness_test.dag @@ -1589,7 +1589,7 @@ test fn witness_fabric_cell_drifted_address_plans_change_and_never_add() -> Bool list_length(items: lines) == 2 && fabric_cell_line_occurrences( lines: lines, - target: "REFUSED-REALIZATION fabric-cell resource-boundary change realization undeclared: no staged intermediates (gunbc.change_realization)", + target: "REFUSED-REALIZATION fabric-cell resource-boundary change has no admitted resource transition (gunbc.change_realization)", ) == 1 && fabric_cell_line_occurrences( lines: lines, @@ -1618,7 +1618,7 @@ test fn witness_fabric_cell_absent_address_plans_add_and_never_change() -> Bool ) == 1 && fabric_cell_line_occurrences( lines: lines, - target: "REFUSED-REALIZATION fabric-cell resource-boundary change realization undeclared: no staged intermediates (gunbc.change_realization)", + target: "REFUSED-REALIZATION fabric-cell resource-boundary change has no admitted resource transition (gunbc.change_realization)", ) == 0 } diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 839e0c3e955..981d3e6541c 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -340,3 +340,23 @@ the actual published ten-slot receipt, including the predecessor's rejecting par All three focused checks pass: the complete host population reaches the workspace consumer, another receipt host refuses, and an unenrolled network slot refuses. Commissioning diagnostics now retain the underlying network/sanitation refusal. + +Controller installation at `f99e440ce59874b10d0c9c65c27a3217a0a1cc3e` passes. +Its commissioning preflight gets through the network receipt and sanitation reads, +then refuses the installed cell boundary. Independent readback finds all five +non-CPU limits correct, but `CPUQuotaPerSecUSec=infinity`; the persistent quota +drop-in still dates to the initial cell apply at 03:47:23. The `MemberAdded` effect +passed the fleet-wide unlimited quota instead of the slot's declared four-CPU quota. +The earlier `FullyApplied` receipt therefore establishes successful commands, +not full resource-boundary convergence. No VM or commissioning mutation ran. + +The add constructor now accepts only the slot and derives its own quota. Existing +state repair admits only an enrolled workspace's unlimited CPU predecessor when +all non-CPU limits already match and the target equals its declared boundary. +`gunbc.change_realization` classifies the preserved address as `InPlaceUpdate`; +the boundary effect emits only `CPUQuota=400%`. Other resource changes, finite predecessors, +quota removal and changed addresses still refuse. This preserves the cell and +its storage; it is not deletion/recreation to make drift look absent. +All thirteen focused cell-effect and fleet-plan checks pass, including the actual +slot-specific add path and the restricted repair. All 8,279 source files parse, +and formatting checks pass. Live repair and commissioning remain outstanding. From 4fa60f7134239a58f91fced3e127e36a5861a303 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 09:08:33 +0000 Subject: [PATCH 15/19] fix(workspace): recognize structural cell slice ancestors --- dag/gunbc/fabric/fabric_cell_observe.dag | 33 ++++- ...y_writer_used_for_reconcilable_staging.dag | 2 +- ...tructural_ancestors_with_foreign_cells.dag | 13 ++ .../fabric_cell_observe_witness_test.dag | 114 ++++++++++++++++-- ...ner_microvm_network_local_stage_probe.dag} | 2 +- docs/plans/srv1-workspace-vm-bringup.md | 21 +++- 6 files changed, 173 insertions(+), 12 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/slice_census_confuses_structural_ancestors_with_foreign_cells.dag rename dag/test/manual/{runner_microvm_network_local_stage_test.dag => runner_microvm_network_local_stage_probe.dag} (98%) diff --git a/dag/gunbc/fabric/fabric_cell_observe.dag b/dag/gunbc/fabric/fabric_cell_observe.dag index a31d718d2b3..8e6c4dcbc43 100644 --- a/dag/gunbc/fabric/fabric_cell_observe.dag +++ b/dag/gunbc/fabric/fabric_cell_observe.dag @@ -3,6 +3,7 @@ module gunbc.fabric_cell_observe import std.types { String, List, Bool, NonEmptyStr } import std.disposition { ConstructionMechanism, SingleAuthority } import product.host_identity { HostIdentity } +import extdeps.systemd { systemd_slice_ancestor_names, systemd_slice_suffix } import gunbc.build_cache_instance { RunnerSlotIdentity, runner_slot_instance_name, runner_slot_identity_equal, runner_slot_instance_name_prefix, slot_name_could_belong_to_host } import gunbc.runner_slot_allocation { fabric_execution_slot_identities, @@ -495,6 +496,7 @@ fn fabric_cell_observe_host(probe: FabricCellHostProbe) -> FabricCellHostReadbac // foreign-artifact remedy. type FabricCellDiscoveredSubject = DiscoveredExpectedCell { slot: RunnerSlotIdentity } + | DiscoveredCellSliceAncestor { observed_name: String, host: HostIdentity } | DiscoveredUnexpectedCommittedSlot { slot: RunnerSlotIdentity } | DiscoveredCrossHostNamedArtifact { observed_name: String, named_slot: RunnerSlotIdentity, observed_on: HostIdentity } | DiscoveredForeignArtifact { observed_name: String, location: String } @@ -503,6 +505,8 @@ type FabricCellDiscoveredSubject fn fabric_cell_discovered_subject_wire(s: FabricCellDiscoveredSubject) -> String { match s { DiscoveredExpectedCell { slot: sl } => join(["expected:", runner_slot_instance_name(slot: sl)], "") + DiscoveredCellSliceAncestor { observed_name: n, host: h } => + join(["cell-slice-ancestor:", h as String, ":", n], "") DiscoveredUnexpectedCommittedSlot { slot: sl } => join(["unexpected-committed:", runner_slot_instance_name(slot: sl)], "") DiscoveredCrossHostNamedArtifact { observed_name: n, named_slot: sl, observed_on: h } => join(["cross-host-named:", h as String, ":", n, ":names:", runner_slot_instance_name(slot: sl)], "") @@ -605,6 +609,21 @@ fn fabric_cell_slot_is_fabric_expected(slot: RunnerSlotIdentity) -> Bool { ) } +// A SLICE NAME ALSO CREATES ITS SYSTEMD ANCESTORS. The namespace enumeration legitimately +// returns fabric-cell-srv1.slice alongside fabric-cell-srv1-13.slice. It is neither a slot nor +// foreign residue. Derive proper ancestors forward from this host's enrolled cell names through +// systemd's existing hierarchy authority; a prefix exclusion would also hide unrelated slices. +// This classification grants no resource ownership or effect, and cannot satisfy a cell address. +fn fabric_cell_is_expected_slice_ancestor(name: String, members: List) -> Bool { + fold(members, init: false, f: (seen, slot) => { + let slice = fabric_cell_slice_name(slot: slot) + let stem = substring(s: slice, start: 0, end: string_length(s: slice) - string_length(s: systemd_slice_suffix)) + seen || (fabric_cell_slot_is_fabric_expected(slot: slot) && name != slice + && fold(systemd_slice_ancestor_names(stem: stem), init: false, f: (matched, ancestor) => + matched || name == join([ancestor, systemd_slice_suffix], ""))) + }) +} + // CLASSIFIED AGAINST THIS HOST'S COMMITTED SLOTS, NOT THE FLEET'S. An earlier cut searched the // complete fleet population with no host argument, so an artifact named `srv3-06` found in srv4's // `/opt/fabric-cells` classified as `DiscoveredExpectedCell { srv3-06 }` -- because that name is @@ -629,7 +648,15 @@ fn fabric_cell_classify_discovered_name( CommittedPopulationDerived { members: members } => fold( fabric_cell_named_among(members: members, name: name, rendering: rendering), - init: fabric_cell_classify_uncommitted_here(host: host, name: name, rendering: rendering), + init: match rendering { + CellRootRendering => fabric_cell_classify_uncommitted_here(host: host, name: name, rendering: rendering) + SliceRendering => + if fabric_cell_is_expected_slice_ancestor(name: name, members: members) { + DiscoveredCellSliceAncestor { observed_name: name, host: host } + } else { + fabric_cell_classify_uncommitted_here(host: host, name: name, rendering: rendering) + } + }, f: (acc, s) => if fabric_cell_slot_is_fabric_expected(slot: s) { DiscoveredExpectedCell { slot: s } @@ -838,7 +865,8 @@ fn fabric_cell_projection_refusals(projection: FabricCellObserverProjection) -> // DISCOVERY PROJECTS INTO THE REFUSAL VOCABULARY gunbc.fabric_cell_converge OWNS, which is what // keeps the direction acyclic: this module imports that one, so that one cannot name -// FabricCellDiscovery. ONLY ONE ARM PROJECTS NOTHING, and an earlier cut got that population wrong +// FabricCellDiscovery. Expected cells and their structural slice ancestors project no refusal; +// ancestors remain explicit discovery subjects but are not cell resources. An earlier cut got that population wrong // in a way worth recording: it also dropped DiscoveredUnexpectedCommittedSlot, on the reasoning that // a committed slot "reaches the classification kernel through the readback". Only EXPECTED slots do // -- fabric_cell_observe_host folds fabric_cell_expected_slots -- so a committed non-fabric slot @@ -868,6 +896,7 @@ fn fabric_cell_discovery_refusals(discovery: FabricCellDiscovery) -> List List { match s { DiscoveredExpectedCell { slot: _ } => [] + DiscoveredCellSliceAncestor { observed_name: _, host: _ } => [] DiscoveredUnexpectedCommittedSlot { slot: sl } => [UnexpectedCommittedSlotArtifact { host: host, slot: sl, location: fabric_cell_base_dir_name() }] DiscoveredCrossHostNamedArtifact { observed_name: n, named_slot: sl, observed_on: h } => diff --git a/dag/gunbc/recurring_failure_mode/create_only_writer_used_for_reconcilable_staging.dag b/dag/gunbc/recurring_failure_mode/create_only_writer_used_for_reconcilable_staging.dag index fef29767b38..f58a0f2e508 100644 --- a/dag/gunbc/recurring_failure_mode/create_only_writer_used_for_reconcilable_staging.dag +++ b/dag/gunbc/recurring_failure_mode/create_only_writer_used_for_reconcilable_staging.dag @@ -8,7 +8,7 @@ data create_only_writer_used_for_reconcilable_staging: RecurringFailureMode = Re receipts: [ "2026-10-09, PR #13607: the local network apply at 682c32644e8 refused File exists for its prior root-owned staging artifacts. No network install operation ran. Filesystem.WriteOwnerOnly creates an exclusive 0600 file; the caller chose it for its custody contract but also assumed replacement semantics it does not provide. A fresh staging directory would hide the defect until the next converge.", "Repair at the write realization: mktemp creates a unique 0600 inode inside the already-verified staging directory, Filesystem.Write fills that inode, and the existing mv_exact_destination_command atomically publishes it on the same filesystem. Scope admission applies to the destination and temporary path. Failures remove only this invocation's temporary inode and report cleanup refusal; final exact byte readback and the outer root-custody readback remain required.", - "Discriminator: test.manual.runner_microvm_network_local_stage_test.local_stage_replaces_owned_bytes_wet executes the real production writer for creation, replacement with different bytes and replay, checks 0600 mode and trailing newlines, refuses a destination outside the supplied scope, and checks no temporary inode remains. This is a manual native filesystem control, not a hermetic CI claim. Rung: external byte/custody facts are observed and refused; constructor-level distinction between exclusive creation and replacement remains the ceiling for callers choosing a filesystem operation. Trigger: operation admission requires the caller's intended publication semantics rather than relying on the writer's name.", + "Discriminator: test.manual.runner_microvm_network_local_stage_probe.local_stage_replaces_owned_bytes_wet executes the real production writer for creation, replacement with different bytes and replay, checks 0600 mode and trailing newlines, refuses a destination outside the supplied scope, and checks no temporary inode remains. This is a manual native filesystem control, not a hermetic CI claim. Rung: external byte/custody facts are observed and refused; constructor-level distinction between exclusive creation and replacement remains the ceiling for callers choosing a filesystem operation. Trigger: operation admission requires the caller's intended publication semantics rather than relying on the writer's name.", ], evidence: [], } diff --git a/dag/gunbc/recurring_failure_mode/slice_census_confuses_structural_ancestors_with_foreign_cells.dag b/dag/gunbc/recurring_failure_mode/slice_census_confuses_structural_ancestors_with_foreign_cells.dag new file mode 100644 index 00000000000..943d4bb17b0 --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/slice_census_confuses_structural_ancestors_with_foreign_cells.dag @@ -0,0 +1,13 @@ +module gunbc.recurring_failure_mode.slice_census_confuses_structural_ancestors_with_foreign_cells + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data slice_census_confuses_structural_ancestors_with_foreign_cells: RecurringFailureMode = RecurringFailureMode { + identity: "slice_census_confuses_structural_ancestors_with_foreign_cells" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607: srv1's systemctl list-units fabric-cell-*.slice enumerates both fabric-cell-srv1-13.slice and its implicit fabric-cell-srv1.slice ancestor. The cell observer assumed every namespace entry was either a slot or foreign residue, so provisioning a legitimate slot made its next observation refuse. Enumeration was correct; the earliest unjustified boundary was discovery's incomplete subject vocabulary.", + "Repair: retain a structural-ancestor subject derived from this host's enrolled fabric cells through extdeps.systemd.systemd_slice_ancestor_names. It grants no cell identity, resource ownership or effect. Only the slice namespace admits it; unknown siblings, other hosts' ancestors and matching filesystem names still refuse. The expected-address observer remains independent, so seeing only the ancestor never satisfies the cell boundary. The live census fixture reaches production admission, with discriminating refusal and conservation controls.", + ], + evidence: [], +} diff --git a/dag/test/claim/fabric/fabric_cell_observe_witness_test.dag b/dag/test/claim/fabric/fabric_cell_observe_witness_test.dag index 1ef5dbad34e..8f93154a6cf 100644 --- a/dag/test/claim/fabric/fabric_cell_observe_witness_test.dag +++ b/dag/test/claim/fabric/fabric_cell_observe_witness_test.dag @@ -3,7 +3,7 @@ module test.claim.fabric_cell_observe_witness import std.types { Bool, String, List, Int, NonEmptyStr, list_length } import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } import gunbc.build_cache_instance { RunnerSlotIdentity, runner_slot_instance_name } -import gunbc.fleet_host_identity { operator_host_srv3, operator_host_srv4 } +import gunbc.fleet_host_identity { operator_host_srv1, operator_host_srv3, operator_host_srv4 } import product.host_identity { HostIdentity } import gunbc.fabric_cell_converge { FabricCellResourceAddress, @@ -62,6 +62,7 @@ import gunbc.fabric_cell_observe { DiscoveryRefused, FabricCellDiscoveredSubject, DiscoveredExpectedCell, + DiscoveredCellSliceAncestor, DiscoveredUnexpectedCommittedSlot, DiscoveredForeignArtifact, DiscoveredHostWidthUnresolved, @@ -482,7 +483,10 @@ fn discovery_wire_set(d: FabricCellDiscovery) -> String { // CLASSIFIES AS SATISFIED THROUGH IT. The end-to-end join: the two halves were built a PR apart and // this is the row that fails if the observer's vocabulary drifts from the kernel's. test fn witness_observer_converged_probe_classifies_as_satisfied() -> Bool { - let readback = fabric_cell_observe_host(probe: converged_probe()) + let readback = FabricCellHostReadback { + host: operator_host_srv3, + cell_readbacks: [fabric_cell_observe_slot(probe: converged_probe(), slot: observe_slot())], + } let classified = classify_fabric_cell_host( expected_slots: [observe_slot()], readback: readback, @@ -661,6 +665,7 @@ test fn witness_observer_other_hosts_committed_name_is_not_this_hosts_expected_c fn discovered_is_expected(s: FabricCellDiscoveredSubject) -> Bool { match s { DiscoveredExpectedCell { slot: _ } => true + DiscoveredCellSliceAncestor { observed_name: _, host: _ } => false DiscoveredUnexpectedCommittedSlot { slot: _ } => false DiscoveredCrossHostNamedArtifact { observed_name: _, named_slot: _, observed_on: _ } => false DiscoveredForeignArtifact { observed_name: _, location: _ } => false @@ -671,6 +676,7 @@ fn discovered_is_expected(s: FabricCellDiscoveredSubject) -> Bool { fn discovered_is_cross_host(s: FabricCellDiscoveredSubject) -> Bool { match s { DiscoveredCrossHostNamedArtifact { observed_name: _, named_slot: _, observed_on: _ } => true + DiscoveredCellSliceAncestor { observed_name: _, host: _ } => false DiscoveredExpectedCell { slot: _ } => false DiscoveredUnexpectedCommittedSlot { slot: _ } => false DiscoveredForeignArtifact { observed_name: _, location: _ } => false @@ -802,7 +808,7 @@ fn population_is_refused(p: FabricCellObservationDecision) -> Bool { // a cell nobody asked about, had no output position and vanished. The observer was conserving what // it PRODUCED while silently dropping part of what it was GIVEN. test fn witness_observer_unconsumed_probe_entries_refuse_rather_than_vanishing() -> Bool { - let stray = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 7 } + let stray = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 99 } let refusals = fabric_cell_unconsumed_probe_refusals(probe: FabricCellHostProbe { host: operator_host_srv3, namespaces: concat( @@ -832,7 +838,7 @@ test fn witness_observer_matching_probe_leaves_nothing_unconsumed() -> Bool { // the projection carries it, and fabric_cell_projection_refusals is the only way to get the // discovery half, so a caller cannot take one and skip the other. test fn witness_observer_unconsumed_probe_refuses_the_population() -> Bool { - let stray = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 7 } + let stray = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 99 } population_is_refused(p: admit_fabric_cell_probe(probe: FabricCellHostProbe { host: operator_host_srv3, namespaces: converged_probe().namespaces, @@ -994,8 +1000,8 @@ fn refusal_identities(host: HostIdentity, entries: List) -> List // the plan body. Two probes describing the same host in a different sequence must not produce two // plan identities. test fn witness_observer_unconsumed_refusals_are_order_independent() -> Bool { - let a = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 7 } - let b = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 8 } + let a = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 99 } + let b = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 100 } let forward = stray_probe_identities(first: a, second: b) let reverse = stray_probe_identities(first: b, second: a) list_length(items: forward) == 2 && identity_lists_equal(a: forward, b: reverse) @@ -1004,7 +1010,7 @@ test fn witness_observer_unconsumed_refusals_are_order_independent() -> Bool { // AND DUPLICATES OF ONE SUBJECT ARE ONE REFUSAL, which is the same population question asked the // other way: an order that is stable but double-counts is no better than one that reorders. test fn witness_observer_duplicate_unconsumed_probes_are_one_refusal() -> Bool { - let a = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 7 } + let a = RunnerSlotIdentity { host: operator_host_srv3, slot_index: 99 } list_length(items: stray_probe_identities(first: a, second: a)) == 1 } @@ -1087,3 +1093,97 @@ test fn witness_observer_refused_slice_enumeration_refuses_discovery() -> Bool { DiscoveryEnumerated { host: _, subjects: _ } => false } } + +// The real srv1 census contains both units after provisioning slot 13. Keep the parent in +// discovery without inventing another cell, and pass the same probe through production admission. +fn srv1_cell_probe(slice_entries: List, root_entries: List) -> FabricCellHostProbe { + let slot = RunnerSlotIdentity { host: operator_host_srv1, slot_index: 13 } + FabricCellHostProbe { + host: operator_host_srv1, + namespaces: [ + enumerated(ns: FabricCellsRootNamespace, entries: root_entries), + enumerated(ns: FabricCellChildNamespace { slot: slot }, entries: ["attempts"]), + enumerated(ns: FabricCellSliceNamespace, entries: slice_entries), + ], + addresses: map(fabric_cell_resource_addresses(), address => FabricCellAddressProbeEntry { + slot: slot, + address: address, + probe: AddressStateRead { state_digest: fabric_cell_desired_state_digest(slot: slot, address: address) }, + }), + } +} + +test fn witness_observer_live_srv1_slice_ancestor_is_retained_and_admitted() -> Bool { + let probe = srv1_cell_probe( + slice_entries: ["fabric-cell-srv1-13.slice", "fabric-cell-srv1.slice"], + root_entries: ["srv1-13"], + ) + match fabric_cell_discovered_subjects(probe: probe) { + DiscoveryRefused { host: _, cause: _ } => false + DiscoveryEnumerated { host: _, subjects: ss } => + join(map(ss, s => fabric_cell_discovered_subject_wire(s: s)), "\n") + == "cell-slice-ancestor:srv1:fabric-cell-srv1.slice\nexpected:srv1-13" + && !population_is_refused(p: admit_fabric_cell_probe(probe: probe)) + } +} + +test fn witness_observer_slice_ancestor_does_not_hide_unknown_siblings() -> Bool { + population_is_refused(p: admit_fabric_cell_probe(probe: srv1_cell_probe( + slice_entries: ["fabric-cell-srv1-13.slice", "fabric-cell-srv1.slice", "fabric-cell-zp444.slice"], + root_entries: ["srv1-13"], + ))) +} + +test fn witness_observer_another_hosts_slice_ancestor_remains_foreign() -> Bool { + let projection = fabric_cell_observe(probe: srv1_cell_probe( + slice_entries: ["fabric-cell-srv1-13.slice", "fabric-cell-srv3.slice"], + root_entries: ["srv1-13"], + )) + match projection.discovery { + DiscoveryRefused { host: _, cause: _ } => false + DiscoveryEnumerated { host: _, subjects: ss } => + fold(ss, init: false, f: (seen, s) => seen || fabric_cell_discovered_subject_wire(s: s) + == "foreign:fabric-slices:fabric-cell-srv3.slice") + && list_length(items: fabric_cell_projection_refusals(projection: projection)) == 1 + } +} + +test fn witness_observer_slice_ancestor_spelling_in_filesystem_still_refuses() -> Bool { + population_is_refused(p: admit_fabric_cell_probe(probe: srv1_cell_probe( + slice_entries: ["fabric-cell-srv1-13.slice", "fabric-cell-srv1.slice"], + root_entries: ["srv1-13", "fabric-cell-srv1.slice"], + ))) +} + +test fn witness_observer_slice_ancestor_cannot_satisfy_the_cell_boundary() -> Bool { + let slot = RunnerSlotIdentity { host: operator_host_srv1, slot_index: 13 } + observation_is_absent(o: fabric_cell_observe_address( + probe: FabricCellHostProbe { + host: operator_host_srv1, + namespaces: [enumerated(ns: FabricCellSliceNamespace, entries: ["fabric-cell-srv1.slice"])], + addresses: [], + }, + slot: slot, + address: FabricCellResourceBoundaryAddress, + )) +} + +test fn witness_observer_slice_ancestors_are_order_independent_and_deduplicated() -> Bool { + let a = fabric_cell_discovered_subjects(probe: srv1_cell_probe( + slice_entries: ["fabric-cell-srv1-13.slice", "fabric-cell-srv1.slice"], root_entries: ["srv1-13"], + )) + let b = fabric_cell_discovered_subjects(probe: srv1_cell_probe( + slice_entries: ["fabric-cell-srv1.slice", "fabric-cell-srv1-13.slice", "fabric-cell-srv1.slice"], root_entries: ["srv1-13"], + )) + match a { + DiscoveryRefused { host: _, cause: _ } => false + DiscoveryEnumerated { host: _, subjects: left } => match b { + DiscoveryRefused { host: _, cause: _ } => false + DiscoveryEnumerated { host: _, subjects: right } => + list_length(items: left) == 2 && identity_lists_equal( + a: map(left, s => fabric_cell_discovered_subject_wire(s: s)), + b: map(right, s => fabric_cell_discovered_subject_wire(s: s)), + ) + } + } +} diff --git a/dag/test/manual/runner_microvm_network_local_stage_test.dag b/dag/test/manual/runner_microvm_network_local_stage_probe.dag similarity index 98% rename from dag/test/manual/runner_microvm_network_local_stage_test.dag rename to dag/test/manual/runner_microvm_network_local_stage_probe.dag index d9f46a196d2..e39e3b81c14 100644 --- a/dag/test/manual/runner_microvm_network_local_stage_test.dag +++ b/dag/test/manual/runner_microvm_network_local_stage_probe.dag @@ -1,4 +1,4 @@ -module test.manual.runner_microvm_network_local_stage_test +module test.manual.runner_microvm_network_local_stage_probe import std.types { String, NonEmptyStr, FilePath, Bool } import std.algebra { trim } diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 981d3e6541c..be3c87e4290 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -308,7 +308,7 @@ The local network staging writer now creates a private temporary inode in the verified directory, fills it, and atomically replaces the exact staged destination. The existing scope admission, root-custody checks and exact byte readback remain required. The manual native filesystem control -`test.manual.runner_microvm_network_local_stage_test.local_stage_replaces_owned_bytes_wet` +`test.manual.runner_microvm_network_local_stage_probe.local_stage_replaces_owned_bytes_wet` passes creation, changed-content replacement, replay, 0600 mode, trailing-newline preservation, out-of-scope refusal and scratch cleanup. The out-of-scope control uses an existing allowed directory, so it distinguishes admission refusal from a @@ -360,3 +360,22 @@ its storage; it is not deletion/recreation to make drift look absent. All thirteen focused cell-effect and fleet-plan checks pass, including the actual slot-specific add path and the restricted repair. All 8,279 source files parse, and formatting checks pass. Live repair and commissioning remain outstanding. + +The read-only cell plan at `ce5bf60b507` records generation 20 and refuses +`fabric-cell-srv1.slice`, the structural parent created by systemd for slot 13. +It was not applied. The namespace census correctly observes both parent and child; +the classifier lacked a subject for proper ancestors. Discovery now derives them +from the local enrolled cell names through the existing systemd hierarchy authority +and retains them without granting ownership or satisfying a cell address. Unrelated +slices and ancestor-looking filesystem entries still refuse. The new controls use +the actual srv1 two-unit census and production observation admission. + +Controller installation and full readback at `ce5bf60b507` pass. CI run +[37903218933](https://github.com/gunb-ai/gunbc/actions/runs/37903218933) passes +seed, emit-build and generated checks, but floor discovery refuses the manual +filesystem probe's `*_test.dag` filename because it declares no enrolled test. +It is renamed `runner_microvm_network_local_stage_probe.dag`; its manual entry and +behavior are unchanged. The observer's first broader batch passes all six new +ancestor controls and exposes four stale single-cell/unenrolled-slot fixtures: +srv3 now enrolls slot 7 too. The isolated classification test now supplies only +its selected slot's readback, and foreign-probe controls use slots 99 and 100. From 0fd9f91c7b1d6623dacfde5427cee9ee0cbb389b Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 09:57:30 +0000 Subject: [PATCH 16/19] fix(workspace): retain the commissioning sanitation refusal --- dag/extdeps/linux/tun_fdinfo.dag | 21 +++++-- ...ing_refusal_discards_the_unproven_fact.dag | 13 ++++ ...ner_microvm_network_sanitation_observe.dag | 4 +- dag/gunbc/workspace_commissioning.dag | 10 +++- dag/gunbc/workspace_commissioning_fleet.dag | 15 +++-- .../workspace_commissioning_witness_test.dag | 22 ++++++- ...orkspace_network_producer_witness_test.dag | 13 +++- docs/plans/srv1-workspace-vm-bringup.md | 60 +++++++++++++++++++ 8 files changed, 140 insertions(+), 18 deletions(-) create mode 100644 dag/gunbc/recurring_failure_mode/commissioning_refusal_discards_the_unproven_fact.dag diff --git a/dag/extdeps/linux/tun_fdinfo.dag b/dag/extdeps/linux/tun_fdinfo.dag index 3b53f33133f..e055231f126 100644 --- a/dag/extdeps/linux/tun_fdinfo.dag +++ b/dag/extdeps/linux/tun_fdinfo.dag @@ -24,10 +24,11 @@ data extdeps_external_authority_anchor: ExternalAuthority = ExternalAuthority { // references held by io_uring/SCM_RIGHTS rather than installed process FDs. Denied reads refuse. // // WHERE THE QUEUE COUNT COMES FROM: the current combined channel count, read through -// extdeps.ethtool. This carries over the earlier inline probe's ETHTOOL_GCHANNELS read, and with it -// that probe's UNVERIFIED assumption that tun answers that request. If it does not, every census is -// Unreadable, which blocks release and never reports zero. The assumption is a live check before -// first commissioning (run `ethtool -l ` against a slot's device on srv1). +// extdeps.ethtool. Linux v6.9 drivers/net/tun.c tun_get_channels publishes tun->numqueues; +// v6.8 has no get_channels callback. srv1's 6.8.0-138-generic returned Operation not supported +// during PR #13607 commissioning on 2026-10-09. An unsupported query remains Unreadable, never +// zero: descriptor sweeps alone miss SCM_RIGHTS/io_uring references. Upgrading a host requires +// a fresh successful capability observation, not admission by a version string. // // ROOT IS ESTABLISHED, NOT INFERRED FROM DENIALS. A non-root reader is refused by the effective uid // on /proc/self/status before anything else is read. Denied fdinfo reads alone would not be @@ -44,6 +45,18 @@ type TunCensusRefusal | TunCensusQueueCountUnreadable { cause: NonEmptyStr } | TunCensusProcUnreadable { path: String } +fn tun_census_refusal_text(cause: TunCensusRefusal) -> String { + match cause { + TunCensusSubjectInadmissible => "TUN census interface name is inadmissible" + TunCensusReaderIdentityUnreadable => "TUN census reader identity is unreadable" + TunCensusReaderNotRoot { euid } => concat("TUN census requires root; effective UID is ", to_string(euid)) + TunCensusFlagsUnreadable => "TUN interface flags are unreadable or malformed" + TunCensusMultiQueue => "TUN census does not admit multi-queue interfaces" + TunCensusQueueCountUnreadable { cause: detail } => concat("TUN attached-queue count is unavailable: ", detail as String) + TunCensusProcUnreadable { path } => concat("TUN descriptor census could not read ", path) + } +} + // An exact nonnegative count must be affirmed; missing or malformed is not zero. `exited` counts // pids and threads that disappeared between being listed and being read. They were skipped, not // treated as holders or as refusals. diff --git a/dag/gunbc/recurring_failure_mode/commissioning_refusal_discards_the_unproven_fact.dag b/dag/gunbc/recurring_failure_mode/commissioning_refusal_discards_the_unproven_fact.dag new file mode 100644 index 00000000000..7964d101ad2 --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/commissioning_refusal_discards_the_unproven_fact.dag @@ -0,0 +1,13 @@ +module gunbc.recurring_failure_mode.commissioning_refusal_discards_the_unproven_fact + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data commissioning_refusal_discards_the_unproven_fact: RecurringFailureMode = RecurringFailureMode { + identity: "commissioning_refusal_discards_the_unproven_fact" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607: the live commissioning plan at 4fa60f71342 passed runtime and store observations, then returned only initial commissioning obligations are not admitted. Its TUN census had already observed ethtool --show-channels gunbc-tap13 fail with Operation not supported on 6.8.0-138-generic. The census cause became a generic TAP refusal, sanitation discarded its unproven facts, and fleet planning reduced the admission sum to a Bool. Three lossy projections hid the failing external capability.", + "The producer now renders the typed census refusal, commissioning uses product.fabric.sanitation.cell_readiness_line, and fleet planning retains the admission refusal. The safety verdict is unchanged: an unavailable kernel queue count cannot become zero, and process descriptor sweeps alone cannot establish absence of SCM_RIGHTS/io_uring-held references. Linux v6.9 drivers/net/tun.c tun_get_channels supplies numqueues; v6.8 lacks that callback. Host capability must be read successfully after any kernel maintenance, not assumed from a version string. Tests distinguish unsupported queue observation and the exact unobservable sanitation fact. Rung: observed refusal with preserved cause, not a guarantee of host capability.", + ], + evidence: [], +} diff --git a/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag b/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag index 2fcf3bde4bf..6fdea2b8ae3 100644 --- a/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag +++ b/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag @@ -4,7 +4,7 @@ import std.types { String, NonEmptyStr, Bool } import std.algebra { trim } import std.resources { Filesystem } import std.content_hash { sha256_hex_digest } -import extdeps.linux.tun_fdinfo { tun_descriptor_census_observe, TunDescriptorCensus, TunDescriptorCount, TunInterfaceMissing, TunDescriptorUnreadable } +import extdeps.linux.tun_fdinfo { tun_descriptor_census_observe, tun_census_refusal_text, TunDescriptorCensus, TunDescriptorCount, TunInterfaceMissing, TunDescriptorUnreadable } import extdeps.exec.command { ArgvCommand, LocalExec } import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent } import extdeps.tools.sha256sum { sha256sum_stdin_digest_via_shell, Sha256FileDigest } @@ -29,7 +29,7 @@ fn tap_holders_reading(census: TunDescriptorCensus) -> TapHolderObservation { match census { TunDescriptorCount { count: n, exited: _ } => if n == 0 { TapPresentUnheld } else { TapHeld { holders: n } } TunInterfaceMissing => TapMissing - TunDescriptorUnreadable { cause: _ } => TapHoldersUnreadable { reason: "root TUN descriptor census unavailable or malformed" as NonEmptyStr } + TunDescriptorUnreadable { cause } => TapHoldersUnreadable { reason: tun_census_refusal_text(cause: cause) as NonEmptyStr } } } diff --git a/dag/gunbc/workspace_commissioning.dag b/dag/gunbc/workspace_commissioning.dag index b15aede4789..3d82c237719 100644 --- a/dag/gunbc/workspace_commissioning.dag +++ b/dag/gunbc/workspace_commissioning.dag @@ -9,7 +9,7 @@ import gunbc.runner_slot_allocation { gunbc_microvm_shakedown_slot_memory_max, g import gunbc.runner_slot_desired { CpuQuotaResolved, SlotCpuQuotaUnbounded } import gunbc.build_cache_instance { RunnerSlotIdentity, runner_slot_instance_name } import gunbc.fleet_host_budget { HostAllocationVerdict, HostAllocationConserves, HostAllocationOversubscribed, HostAllocationUnchargeable } -import product.fabric.sanitation { CellId, CellReady, CellQuarantined, SanitationObservation, SlotScopedNetwork, cell_readiness_from_observations } +import product.fabric.sanitation { CellId, CellReady, CellQuarantined, SanitationObservation, SlotScopedNetwork, cell_readiness_from_observations, cell_readiness_line } // This is administrator commissioning intent bound to an existing convergence artifact. It is // not inferred from a missing directory or a failed unit. No current observation mints authorization; the effect may select WIF or app approval. @@ -69,8 +69,12 @@ fn workspace_commissioning_admit( WorkspaceObligationHistoryUnreadable { reason: detail } => WorkspaceCommissioningRefused { reason: detail } WorkspaceNoPriorObligations { settled_generation: generation } => if generation != 0 { WorkspaceCommissioningRefused { reason: "initial commissioning requires an observed never-recorded slot; nonzero history must be recovered" } - } else { match cell_readiness_from_observations(cell: runner_slot_instance_name(slot: slot) as CellId, network_scope: SlotScopedNetwork, observations: sanitation) { - CellQuarantined { cell: _, unproven: _ } => WorkspaceCommissioningRefused { reason: "initial cell sanitation is incomplete; no initial readiness may be published" } + } else { + let readiness = cell_readiness_from_observations(cell: runner_slot_instance_name(slot: slot) as CellId, network_scope: SlotScopedNetwork, observations: sanitation) + match readiness { + CellQuarantined { cell: _, unproven: _ } => WorkspaceCommissioningRefused { + reason: concat("initial cell sanitation is incomplete; no initial readiness may be published: ", cell_readiness_line(readiness: readiness)), + } CellReady { cell: _ } => WorkspaceCommissioningReady { commissioned: WorkspaceCommissionedSlot { slot: slot, cpu_budget: cpu_budget, attempt_envelope: attempt_envelope, commissioning_plan_hash: approved_plan, settled_generation: generation, diff --git a/dag/gunbc/workspace_commissioning_fleet.dag b/dag/gunbc/workspace_commissioning_fleet.dag index fb09ea3b730..49a413c006d 100644 --- a/dag/gunbc/workspace_commissioning_fleet.dag +++ b/dag/gunbc/workspace_commissioning_fleet.dag @@ -170,21 +170,24 @@ fn workspace_commissioning_basis_observe(provenance: WorkspaceCommissioningProve } else if !committed && (match workspace_initial_controller_history_refusal(slot: slot) { Present { value: _ } => true Absent => false }) { WorkspaceCommissioningFleetRefused { reason: "prior controller history requires recovery" } } else { - let admitted = committed || (match workspace_commissioning_admit(slot: slot, + let admission_refusal = if committed { none } else { match workspace_commissioning_admit(slot: slot, authority: WorkspaceCommissioningAuthorizedPlan { slot: slot, plan_hash: provenance.plan_hash }, budget: HostAllocationConserves, cpu_budget: provenance.cpu_budget, attempt_envelope: provenance.attempt_envelope, prior_purpose: WorkspacePriorPurposeWithdrawn, obligations: workspace_observe_commissioning_obligations(root: fabric_control_plane_store_root(), slot: slot), sanitation: observation.observations) { - WorkspaceCommissioningReady { commissioned: _ } => true - WorkspaceCommissioningRefused { reason: _ } => false - }) - if admitted { WorkspaceCommissioningFleetObserved { basis: WorkspaceCommissioningFleetBasis { + WorkspaceCommissioningReady { commissioned: _ } => none + WorkspaceCommissioningRefused { reason } => Present { value: reason } + } } + match admission_refusal { + Absent => WorkspaceCommissioningFleetObserved { basis: WorkspaceCommissioningFleetBasis { provenance: provenance, release_dir: installed, already_committed: committed, prestate: serialize_content_hash(hash: content_hash_of_value(value: to_string([ to_string(snapshot.expected), to_string(snapshot.reading), to_string(execution.expected), to_string(readiness), if initial { to_string(HoldSlotAbsent) } else { to_string(mutation) }, to_string(controller), observation.observation_identity as String, ]) as NonEmptyStr)) as NonEmptyStr, - } } } else { WorkspaceCommissioningFleetRefused { reason: "initial commissioning obligations are not admitted" } } + } } + Present { value: reason } => WorkspaceCommissioningFleetRefused { reason: reason } + } } WorkspaceCommissioningStartUnreadable { reason: _ } => WorkspaceCommissioningFleetRefused { reason: "controller prestate is unreadable" } } } diff --git a/dag/test/claim/workspace_commissioning_witness_test.dag b/dag/test/claim/workspace_commissioning_witness_test.dag index 6d50795ec38..f1ccda54bc8 100644 --- a/dag/test/claim/workspace_commissioning_witness_test.dag +++ b/dag/test/claim/workspace_commissioning_witness_test.dag @@ -13,7 +13,7 @@ import product.host_identity { HostIdentity } import gunbc.runner_slot_allocation { gunbc_microvm_shakedown_slot_memory_max, gunbc_runner_slot_desired_for } import gunbc.runner_slot_desired { SlotCpuQuotaUnbounded } import gunbc.fleet_host_budget { HostAllocationConserves } -import product.fabric.sanitation { required_sanitation_facts, SlotScopedNetwork, FactConfirmed } +import product.fabric.sanitation { required_sanitation_facts, SlotScopedNetwork, SlotNetworkQuiescent, FactConfirmed, FactUnobservable } import gunbc.runner_microvm_cell_readiness { WriteCommissioned, apply_cell_readiness_write, WriteAccepted } fn commissioning_test_slot() -> RunnerSlotIdentity { RunnerSlotIdentity { host: "srv1" as HostIdentity, slot_index: 13 } } @@ -84,3 +84,23 @@ test fn workspace_cpu_binding_does_not_cap_another_slot() -> Bool { _ => false } } + +test fn commissioning_refusal_names_the_unobservable_sanitation_fact() -> Bool { + let slot = commissioning_test_slot() + match workspace_commissioning_admit(slot: slot, + authority: WorkspaceCommissioningAuthorizedPlan { slot: slot, plan_hash: "fixture-plan" as NonEmptyStr }, + budget: HostAllocationConserves, cpu_budget: hardware_thread_count(count: 4), + attempt_envelope: gunbc_microvm_shakedown_slot_memory_max, + prior_purpose: WorkspacePriorPurposeWithdrawn, + obligations: WorkspaceNoPriorObligations { settled_generation: 0 }, + sanitation: map(required_sanitation_facts(scope: SlotScopedNetwork), fact => + if fact == SlotNetworkQuiescent { + FactUnobservable { fact: fact, reason: "tap-unheld: TUN attached-queue count is unavailable" as NonEmptyStr } + } else { FactConfirmed { fact: fact } }), + ) { + WorkspaceCommissioningRefused { reason } => + string_contains(s: reason, pattern: "slot-network-quiescent") + && string_contains(s: reason, pattern: "tap-unheld: TUN attached-queue count is unavailable") + other => false + } +} diff --git a/dag/test/claim/workspace_network_producer_witness_test.dag b/dag/test/claim/workspace_network_producer_witness_test.dag index 7269b53b761..0414bc0f3a1 100644 --- a/dag/test/claim/workspace_network_producer_witness_test.dag +++ b/dag/test/claim/workspace_network_producer_witness_test.dag @@ -1,13 +1,13 @@ module test.claim.workspace_network_producer_witness_test -import std.types { Bool } +import std.types { Bool, NonEmptyStr } import gunbc.runner_microvm_network_observe { network_read_outcome } import gunbc.host_effect { SshShell } import gunbc.executor_privileged_operation { ReadNftRuleset } import extdeps.shell.exec { ProcessRefused } import gunbc.runner_microvm_network_sanitation_observe { tap_holders_reading } import gunbc.runner_microvm_network { TapPresentUnheld, TapHeld, TapHoldersUnreadable } -import extdeps.linux.tun_fdinfo { TunDescriptorCount, TunDescriptorUnreadable, TunCensusMultiQueue } +import extdeps.linux.tun_fdinfo { TunDescriptorCount, TunDescriptorUnreadable, TunCensusMultiQueue, TunCensusQueueCountUnreadable } test fn network_producer_requires_successful_tun_census() -> Bool { (match tap_holders_reading(census: TunDescriptorCount { count: 0, exited: 0 }) { TapPresentUnheld => true other => false }) @@ -15,6 +15,15 @@ test fn network_producer_requires_successful_tun_census() -> Bool { && (match tap_holders_reading(census: TunDescriptorUnreadable { cause: TunCensusMultiQueue }) { TapHoldersUnreadable { reason: _ } => true other => false }) } +test fn network_producer_preserves_unsupported_queue_observation() -> Bool { + match tap_holders_reading(census: TunDescriptorUnreadable { + cause: TunCensusQueueCountUnreadable { cause: "ethtool --show-channels exited nonzero" as NonEmptyStr }, + }) { + TapHoldersUnreadable { reason } => reason == "TUN attached-queue count is unavailable: ethtool --show-channels exited nonzero" + other => false + } +} + test fn unanchored_network_observation_refuses_before_opening_a_session() -> Bool { match network_read_outcome(transport: SshShell { ssh_host: "unexpected-host" }, op: ReadNftRuleset) { diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index be3c87e4290..aa7639dc429 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -379,3 +379,63 @@ behavior are unchanged. The observer's first broader batch passes all six new ancestor controls and exposes four stale single-cell/unenrolled-slot fixtures: srv3 now enrolls slot 7 too. The isolated classification test now supplies only its selected slot's readback, and foreign-probe controls use slots 99 and 100. + +At `4fa60f7134239a58f91fced3e127e36a5861a303`, all 52 observer checks pass, +all 8,280 source files parse, and formatting passes. Controller installation and +full readback pass. The fresh cell plan `local-2026-10-09T09:17:53Z`, hash +`e3dc463bbc6f76f8`, admits only `CPUQuota=400%`, a start of the existing slice, +and the existing root-owned base-directory ensure. Its apply receipt +`43cf1e08f8680971` records exit 0 and generation 19 → 20. Independent systemd +readback confirms `CPUQuotaPerSecUSec=4s`, `MemoryHigh=MemoryMax=30064771072`, +`MemorySwapMax=0`, `TasksMax=16384`, `CPUWeight=100`, and an active slice. +The commissioning plan against this verified release refuses as detailed below; +no VM boot or guest login is yet claimed. CI run +[37910533190](https://github.com/gunb-ai/gunbc/actions/runs/37910533190) covers +this pushed revision. + +The commissioning plan at `4fa60f71342` passes installed-runtime checks but +refuses initial sanitation. Its TUN holder census invokes +`ethtool --show-channels gunbc-tap13`, which returns `Operation not supported` +on srv1's running `6.8.0-138-generic`. The installed `6.8.0-142-generic` +System.map also lacks `tun_get_channels` (while naming `tun_fill_info` and +`tun_ethtool_ops`), so the pending ordinary reboot does not establish this capability. +The original refusal was hidden by three projections; the census cause, unproven +sanitation fact and commissioning admission reason are now retained end to end. + +[Linux v6.8's TUN driver](https://github.com/torvalds/linux/blob/v6.8/drivers/net/tun.c) +has no channel-count callback. [Linux v6.9's implementation](https://github.com/torvalds/linux/blob/v6.9/drivers/net/tun.c#L3442) +reports the attached queue count. Descriptor sweeps alone cannot substitute for it, +because file references can remain outside process descriptor tables. The v6.8 +netlink queue counters are emitted only for multi-queue devices; changing this TAP +to multi-queue would not match [Firecracker v1.16.1's open flags](https://github.com/firecracker-microvm/firecracker/blob/v1.16.1/src/vmm/src/devices/virtio/net/tap.rs#L109). +No refusal has been relaxed, no readiness record has been fabricated, and no VM +has been commissioned or booted. + +### Proposed host maintenance (not applied) + +The read-only package simulation on srv1 selects Ubuntu's arm64 HWE meta-package +`linux-generic-hwe-24.04=7.0.0-38.38~24.04.4`. It adds eight packages, upgrades +none and removes none: the image, modules, ZFS module package, header packages +and HWE image/header meta-packages. This leaves the existing 6.8 kernels installed. +The downloaded Ubuntu image/modules packages were inspected without installation. +The modules package's `System.map-7.0.0-38-generic` contains `tun_get_channels`, +`tun_fill_info` and `tun_ethtool_ops`; this checks the proposed binary's symbols, +not merely its version. The simulated install is: + +```sh +sudo apt-get --no-install-recommends install linux-generic-hwe-24.04=7.0.0-38.38~24.04.4 +``` + +This requires separate operator approval for shared-host maintenance: the reboot +interrupts srv1's twelve runner services, dashboard, fabric storage, approval +broker, ntfy and container services. Before installing or rebooting, drain work, +establish console/boot recovery, and recheck the package plan and disk capacity. +GRUB currently boots entry 0 with a hidden, zero-second menu; keeping an old +kernel alone is not automatic rollback. After reboot, require successful +`ethtool --show-channels gunbc-tap13`, verify KVM, network and service recovery, +then rerun the commissioning plan. A newer version string alone is insufficient. +Local read-only IPMI device/chassis queries succeed and SOL reports enabled; +remote console access and boot recovery have not been established. The current +GitHub credential cannot list organization runners (403), so a runner drain must +be established separately rather than inferred from the repository runner list. +No kernel package has been installed and no reboot has been initiated. From 1ae94ddba7a94105f7cf1403b69d6c571d46978a Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 10:04:46 +0000 Subject: [PATCH 17/19] docs(workspace): record verified kernel blocker and maintenance proposal --- docs/plans/srv1-workspace-vm-bringup.md | 28 ++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index aa7639dc429..f7ce1ccb2b4 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -4,6 +4,12 @@ Scope: the operator-owned workspace on `srv1-13`, using the existing commissioni allocation, controller and cleanup protocols. No customer create API, additional hosts, CI-runner migration, or automatic expiry scheduling is part of this change. +Current status: the pinned image/controller, network receipt and slot resource +boundary have passed live readback. Commissioning refuses because srv1's running +kernel cannot report attached TUN queues. A concrete kernel maintenance proposal +is recorded below and has not been applied. No VM boot or authenticated guest SSH +is yet proven. Guest access uses the dedicated key already on srv1. + ## Observed gaps The October 9, 2026 inspection found no successful workspace VM lifecycle receipt. @@ -420,7 +426,7 @@ and HWE image/header meta-packages. This leaves the existing 6.8 kernels install The downloaded Ubuntu image/modules packages were inspected without installation. The modules package's `System.map-7.0.0-38-generic` contains `tun_get_channels`, `tun_fill_info` and `tun_ethtool_ops`; this checks the proposed binary's symbols, -not merely its version. The simulated install is: +not merely its version. The install command whose simulation was reviewed is: ```sh sudo apt-get --no-install-recommends install linux-generic-hwe-24.04=7.0.0-38.38~24.04.4 @@ -439,3 +445,23 @@ remote console access and boot recovery have not been established. The current GitHub credential cannot list organization runners (403), so a runner drain must be established separately rather than inferred from the repository runner list. No kernel package has been installed and no reboot has been initiated. + +The diagnostics repair at `0fd9f91c7b1` passes all 29 focused commissioning, +network-producer, TUN-census and initial-sanitation witnesses in +`target/srv1-commissioning-diagnostics-tests.log` (exit 0). All 8,281 source files +parse; formatting and whitespace checks pass. The new controls require the actual +queue-query cause and unobservable sanitation fact to survive refusal reporting. + +A read-only manual invocation of the production `workspace_dispatch_plan_cli` +from clean source `0fd9f91c7b1`, using the installed root-owned interpreter and +observing release `4fa60f71342`, exits 1 with the complete live cause: + +```text +privileged commissioning observation refused: initial cell sanitation is incomplete; no initial readiness may be published: srv1-13: quarantined — slot-network-quiescent: unobservable — tap-unheld: TUN attached-queue count is unavailable: ethtool --show-channels exited nonzero +``` + +The captured log is `target/srv1-bringup-evidence/commissioning-readback-0fd9f91.log`. +This validates refusal reporting against the installed release; it is not a new +controller installation, commissioning apply or ongoing CI claim. Independent +post-readback checks find no allocation, commissioning or readiness files, and +the historical controller remains stopped with `MainPID=0` and no queued job. From 187e0eb89a8a2fbda6a0cae265aedddfc5b8ecfb Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 15:41:42 +0000 Subject: [PATCH 18/19] Derive host kernel prerequisites from workspace sanitation --- dag/extdeps/cadquery/cadquery.dag | 3 +- dag/extdeps/ethtool/ethtool.dag | 89 ---------- dag/extdeps/exec/command.dag | 2 + dag/extdeps/exec/program.dag | 2 +- dag/extdeps/linux/ethtool_ioctl.dag | 90 ++++++++++ dag/extdeps/linux/tun_fdinfo.dag | 39 ++++- dag/extdeps/package_managers/apt.dag | 16 ++ dag/extdeps/package_managers/dpkg.dag | 14 ++ dag/extdeps/python/python.dag | 4 + dag/extdeps/tools/uname.dag | 4 + dag/gunbc/fleet/fleet_converge_plan.dag | 14 ++ dag/gunbc/fleet/fleet_converge_plan_cli.dag | 16 +- dag/gunbc/fleet/fleet_converge_workflow.dag | 19 ++- dag/gunbc/host/host_kernel_dependency.dag | 157 ++++++++++++++++++ .../host/host_kernel_dependency_observe.dag | 125 ++++++++++++++ .../host/host_kernel_dependency_wire.dag | 81 +++++++++ ...nal_equality_answers_by_representation.dag | 2 + ...tics_instead_of_demanding_its_provider.dag | 14 ++ ...ner_microvm_network_sanitation_observe.dag | 11 +- .../workspace_commissioning_dispatch.dag | 5 +- ...pace_commissioning_executor_submission.dag | 3 +- dag/gunbc/workspace_commissioning_fleet.dag | 41 ++++- .../host_kernel_dependency_witness_test.dag | 145 ++++++++++++++++ ...space_network_observation_witness_test.dag | 29 ++-- ...space_network_real_reader_witness_test.dag | 5 + ...rkspace_workflow_dispatch_witness_test.dag | 9 +- docs/plans/srv1-workspace-vm-bringup.md | 77 ++++++++- 27 files changed, 875 insertions(+), 141 deletions(-) create mode 100644 dag/extdeps/linux/ethtool_ioctl.dag create mode 100644 dag/gunbc/host/host_kernel_dependency.dag create mode 100644 dag/gunbc/host/host_kernel_dependency_observe.dag create mode 100644 dag/gunbc/host/host_kernel_dependency_wire.dag create mode 100644 dag/gunbc/recurring_failure_mode/unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider.dag create mode 100644 dag/test/claim/host_kernel_dependency_witness_test.dag diff --git a/dag/extdeps/cadquery/cadquery.dag b/dag/extdeps/cadquery/cadquery.dag index 3c2173319ae..d7b5212778e 100644 --- a/dag/extdeps/cadquery/cadquery.dag +++ b/dag/extdeps/cadquery/cadquery.dag @@ -1,6 +1,6 @@ module extdeps.cadquery.cadquery -import extdeps.exec.program { ProgramIdentity, uncataloged_program } +import extdeps.python { python_program } import std.types { Bool, String, NonEmptyStr, List } import std.measure { Micrometer, micrometer_count } @@ -41,7 +41,6 @@ data cadquery_ocp_distribution: OcpDistribution = OcpWithVtk // PATH-resolved, for the reason extdeps.exec.command program_spelling_is_an_observation_claim // records: an absolute path is a claim about a filesystem this repository has not observed. -fn python_program() -> ProgramIdentity = uncataloged_program(invocation: "python3") // RUN ONE EMITTED PROGRAM. The script path is the caller's -- this module has no authority to // enumerate the programs a consumer might run, exactly as extdeps.tools.make refuses to enumerate diff --git a/dag/extdeps/ethtool/ethtool.dag b/dag/extdeps/ethtool/ethtool.dag index 909dc3409dc..80a84073dd7 100644 --- a/dag/extdeps/ethtool/ethtool.dag +++ b/dag/extdeps/ethtool/ethtool.dag @@ -98,92 +98,3 @@ fn ethtool_module_info_raw_command(device: NonEmptyStr) -> NonEmptyStr { " ", ethtool_module_info_raw_on as String, ], "") as NonEmptyStr } - -// ================= THE CHANNEL READ ================= -// -// `-l` (long form `--show-channels`) reports the device's channel counts: ethtool issues -// ETHTOOL_MSG_CHANNELS_GET over netlink, or ETHTOOL_GCHANNELS where netlink is unavailable. Both -// answer the same `struct ethtool_channels` (include/uapi/linux/ethtool.h). The rendering cited here -// is netlink/channels.c in ethtool 6.7, the version measured on spark-a3ee (see -// ethtool_binary_directory): a header line `Channel parameters for :`, then a `Pre-set -// maximums:` block and a `Current hardware settings:` block, each holding `RX:`, `TX:`, `Other:` and -// `Combined:` lines. A count the driver does not report is printed as `n/a`, not as 0. -// -// THE CURRENT COMBINED COUNT IS THE ONLY FIELD READ, because it is the one any consumer here needs: -// struct ethtool_channels.combined_count, the field the earlier inline ioctl probe read as -// channels[8]. Every other outcome is a refusal and never a count. That covers a nonzero exit (the -// driver has no get_channels, or the device is gone), a missing header, a missing current block, a -// missing `Combined:` line in that block, and `n/a` or any other non-decimal value. A refusal that -// fell back to 0 would make an unreadable count look like an idle device. -data ethtool_show_channels_flag: NonEmptyStr = "--show-channels" as NonEmptyStr - -service ethtool.Ethtool { - operation ShowChannels { - input { device: NonEmptyStr } - output { value: String from "stdout", success: Bool from "exit_success" } - readonly - transport shell { argv: ["/usr/sbin/ethtool", "--show-channels", "{device}"] } - exit { 0 => Unit nonzero => String "ethtool channel read failed" } - mock_response { 1 => { value: "", success: false } "no fabricated channel parameters" } - } -} - -type EthtoolCombinedChannels - = EthtoolCombinedChannelsCurrent { count: Int } - | EthtoolCombinedChannelsUnreadable { cause: NonEmptyStr } - -fn ethtool_decimal_value(text: String) -> Int? { - let cps = text |> chars - if length(cps) == 0 || length(cps) > 10 || !all(cps, cp => cp >= 48 && cp <= 57) { none } - else { parse_int(s: text) } -} - -fn ethtool_channel_line_value(line: String, label: String) -> String? { - if starts_with(s: line, prefix: label) { - Present { value: trim(s: substring(s: line, start: string_length(s: label), end: string_length(s: line))) } - } else { none } -} - -// The current `Combined:` line is the first one AFTER the `Current hardware settings:` line, so the -// pre-set maximum, which carries the same label, can never be read in its place. -fn ethtool_current_combined_from(lines: List) -> EthtoolCombinedChannels { - let after = fold(lines, init: EthtoolChannelScanBefore {}, f: (acc, line) => match acc { - EthtoolChannelScanBefore {} => - if trim(s: line) == "Current hardware settings:" { EthtoolChannelScanCurrent {} } else { acc } - EthtoolChannelScanCurrent {} => match ethtool_channel_line_value(line: line, label: "Combined:") { - Present { value: v } => EthtoolChannelScanFound { value: v } - Absent => acc - } - EthtoolChannelScanFound { value: _ } => acc - }) - match after { - EthtoolChannelScanBefore {} => EthtoolCombinedChannelsUnreadable { cause: "no current hardware settings block" as NonEmptyStr } - EthtoolChannelScanCurrent {} => EthtoolCombinedChannelsUnreadable { cause: "no current Combined count" as NonEmptyStr } - EthtoolChannelScanFound { value: v } => match ethtool_decimal_value(text: v) { - Present { value: n } => EthtoolCombinedChannelsCurrent { count: n } - Absent => EthtoolCombinedChannelsUnreadable { cause: "current Combined count is not a decimal count" as NonEmptyStr } - } - } -} - -type EthtoolChannelScan - = EthtoolChannelScanBefore {} - | EthtoolChannelScanCurrent {} - | EthtoolChannelScanFound { value: String } - -fn ethtool_combined_channels(device: NonEmptyStr, success: Bool, value: String) -> EthtoolCombinedChannels { - let lines = split(s: value, delimiter: "\n") - if !success { EthtoolCombinedChannelsUnreadable { cause: "ethtool --show-channels exited nonzero" as NonEmptyStr } } - else { match lines |> first { - Absent => EthtoolCombinedChannelsUnreadable { cause: "empty channel report" as NonEmptyStr } - Present { value: header } => - if trim(s: header) != join(["Channel parameters for ", device as String, ":"], "") { - EthtoolCombinedChannelsUnreadable { cause: "channel report names a different device or no device" as NonEmptyStr } - } else { ethtool_current_combined_from(lines: lines) } - } } -} - -fn ethtool_observe_combined_channels(device: NonEmptyStr) -> EthtoolCombinedChannels { - let read = ethtool.Ethtool.ShowChannels(device: device) - ethtool_combined_channels(device: device, success: read.success, value: read.value) -} diff --git a/dag/extdeps/exec/command.dag b/dag/extdeps/exec/command.dag index 4041e38d681..2b82e203bcc 100644 --- a/dag/extdeps/exec/command.dag +++ b/dag/extdeps/exec/command.dag @@ -136,6 +136,8 @@ fn argv_command(program: ProgramIdentity, arguments: List) -> ArgvComman decl_ref(module_path: "extdeps.iproute2.ip_show", decl_name: "ip_addr_show_command"), decl_ref(module_path: "extdeps.iproute2.ip_show", decl_name: "ip_route_show_command"), decl_ref(module_path: "extdeps.tools.uname", decl_name: "uname_release_command"), + decl_ref(module_path: "extdeps.tools.uname", decl_name: "uname_machine_command"), + decl_ref(module_path: "extdeps.linux.ethtool_ioctl", decl_name: "linux_ethtool_channels_command"), decl_ref(module_path: "extdeps.tools.make", decl_name: "make_command"), decl_ref(module_path: "extdeps.systemd.systemd_run", decl_name: "systemd_run_command_of"), decl_ref(module_path: "extdeps.package_managers.pip", decl_name: "pip_install_pinned_command"), diff --git a/dag/extdeps/exec/program.dag b/dag/extdeps/exec/program.dag index b3b316d7d6f..cc228f87ffd 100644 --- a/dag/extdeps/exec/program.dag +++ b/dag/extdeps/exec/program.dag @@ -166,7 +166,7 @@ fn uncataloged_program(invocation: NonEmptyStr) -> ProgramIdentity decl_ref(module_path: "extdeps.storage.nbd", decl_name: "nbd_server_program"), decl_ref(module_path: "extdeps.package_managers.pip", decl_name: "pip_program"), decl_ref(module_path: "extdeps.posix.sh_invocation", decl_name: "posix_sh_program"), - decl_ref(module_path: "extdeps.cadquery.cadquery", decl_name: "python_program"), + decl_ref(module_path: "extdeps.python", decl_name: "python_program"), decl_ref(module_path: "extdeps.dpkg", decl_name: "dpkg_query_status_command"), decl_ref(module_path: "extdeps.ssh.openssh_client_commands", decl_name: "scp_program"), decl_ref(module_path: "extdeps.ssh.openssh_client_commands", decl_name: "ssh_add_program"), diff --git a/dag/extdeps/linux/ethtool_ioctl.dag b/dag/extdeps/linux/ethtool_ioctl.dag new file mode 100644 index 00000000000..ae1590ed022 --- /dev/null +++ b/dag/extdeps/linux/ethtool_ioctl.dag @@ -0,0 +1,90 @@ +module extdeps.linux.ethtool_ioctl + +import std.types { String, NonEmptyStr, Int, List, Bool } +import std.algebra { trim } +import extdeps.external_authority { ExternalAuthority } +import extdeps.uri { Uri, Https } +import extdeps.python { PythonProgramText, python_run_program_argv, python_program } +import extdeps.exec.command { ArgvCommand, argv_command, LocalExec } +import gunbc.command_runner { run_shell_command_observe_outcome } +import extdeps.shell.exec { ProcessRan, ProcessNotSpawned } + +data extdeps_external_authority_anchor: ExternalAuthority = ExternalAuthority { + uri: Uri { scheme: Https, locator: "github.com/torvalds/linux/blob/v6.9/include/uapi/linux/ethtool.h" } +} + +// Linux's ETHTOOL_GCHANNELS/SIOCETHTOOL ABI, not ethtool's diagnostic prose. The Python +// realization only marshals the native ABI and returns the syscall's errno or all nine u32s. +// No package selection or repair policy lives in it. PythonProgramText carries the same +// declared second-medium boundary as extdeps.nvidia.cuda_driver. This realization admits +// Linux's 64-bit ifreq layout (16-byte name, 24-byte union); other ABIs fail before ioctl. +// include/uapi/linux/{ethtool.h,sockios.h,if.h}; include/uapi/asm-generic/errno.h. +data linux_siocethtool: Int = 35142 +data linux_ethtool_gchannels: Int = 60 +data linux_eopnotsupp: Int = 95 + +type LinuxEthtoolChannels + = LinuxEthtoolChannelsRead { combined: Int } + | LinuxEthtoolChannelsUnsupported + | LinuxEthtoolChannelsUnreadable { cause: NonEmptyStr } + +fn linux_ethtool_channels_program() -> PythonProgramText { + join([ + "import array, fcntl, socket, struct, sys", + "name = sys.argv[1]", + "assert struct.calcsize('P') == 8 and array.array('I').itemsize == 4", + "encoded = name.encode('ascii')", + "assert 0 < len(encoded) < 16 and chr(0) not in name", + concat("channels = array.array('I', [", concat(to_string(linux_ethtool_gchannels), "] + [0] * 8)")), + "with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:", + " request = struct.pack('16sP', encoded, channels.buffer_info()[0]).ljust(40, bytes([0]))", + " try:", + concat(" fcntl.ioctl(sock.fileno(), ", concat(to_string(linux_siocethtool), ", request)")), + " except OSError as error:", + " print('errno:' + name + ':' + str(error.errno))", + " else:", + " print('channels:' + name + ':' + ':'.join(str(x) for x in channels))", + ], "\n") as PythonProgramText +} + +fn linux_ethtool_channels_command(device: NonEmptyStr) -> ArgvCommand { + let words = python_run_program_argv(program: linux_ethtool_channels_program()) + argv_command(program: python_program(), arguments: words.skip(n: 1) + [device as String]) +} + +// Eliminate first() by a match, following optional_equality_answers_by_representation: +// the seed's raw optional carrier cannot be compared with a Present literal. +fn linux_ethtool_channels_decode(device: NonEmptyStr, text: String) -> LinuxEthtoolChannels { + let fields = split(s: trim(s: text), delimiter: ":") + let prefix = concat("channels:", concat(device as String, ":")) + if trim(s: text) == concat("errno:", concat(device as String, concat(":", to_string(linux_eopnotsupp)))) { + LinuxEthtoolChannelsUnsupported + } else if fields.count() == 11 && starts_with(s: trim(s: text), prefix: prefix) { + let numbers = fields.skip(n: 2) + let valid = all(numbers, word => string_length(s: word) > 0 && string_length(s: word) <= 10 + && all(chars(word), cp => cp >= 48 && cp <= 57) + && (match parse_int(s: word) { Present { value: n } => n >= 0 && n <= 4294967295 Absent => false })) + let command_matches = match numbers.first() { + Present { value: command } => command == to_string(linux_ethtool_gchannels) + Absent => false + } + if !valid || !command_matches { + LinuxEthtoolChannelsUnreadable { cause: "ETHTOOL_GCHANNELS returned malformed fields or another command" as NonEmptyStr } + } else { match numbers.last() { + Present { value: last } => match parse_int(s: last) { + Present { value: count } => LinuxEthtoolChannelsRead { combined: count } + Absent => LinuxEthtoolChannelsUnreadable { cause: "ETHTOOL_GCHANNELS combined count is malformed" as NonEmptyStr } + } + Absent => LinuxEthtoolChannelsUnreadable { cause: "ETHTOOL_GCHANNELS combined count is missing" as NonEmptyStr } + } } + } else { LinuxEthtoolChannelsUnreadable { cause: concat("ETHTOOL_GCHANNELS read refused: ", trim(s: text)) as NonEmptyStr } } +} + +fn linux_ethtool_channels_observe(device: NonEmptyStr) -> LinuxEthtoolChannels { + match run_shell_command_observe_outcome(command: linux_ethtool_channels_command(device: device), transport: LocalExec) { + ProcessRan { observation } => if observation.exit_code == 0 { + linux_ethtool_channels_decode(device: device, text: observation.stdout) + } else { LinuxEthtoolChannelsUnreadable { cause: "ETHTOOL_GCHANNELS reader could not execute successfully" as NonEmptyStr } } + ProcessNotSpawned { program: _, cause } => LinuxEthtoolChannelsUnreadable { cause: cause } + } +} diff --git a/dag/extdeps/linux/tun_fdinfo.dag b/dag/extdeps/linux/tun_fdinfo.dag index e055231f126..464e6633ef9 100644 --- a/dag/extdeps/linux/tun_fdinfo.dag +++ b/dag/extdeps/linux/tun_fdinfo.dag @@ -12,7 +12,7 @@ import extdeps.filesystem.filesystem_io { } import extdeps.linux.procfs { procfs_path_literal, ProcSelfStatus, procfs_status_effective_uid } import extdeps.numeric.base16 { base16_word_value } -import extdeps.ethtool { ethtool_observe_combined_channels, EthtoolCombinedChannels, EthtoolCombinedChannelsCurrent, EthtoolCombinedChannelsUnreadable } +import extdeps.linux.ethtool_ioctl { linux_ethtool_channels_observe, LinuxEthtoolChannels, LinuxEthtoolChannelsRead, LinuxEthtoolChannelsUnsupported, LinuxEthtoolChannelsUnreadable } data extdeps_external_authority_anchor: ExternalAuthority = ExternalAuthority { uri: Uri { scheme: Https, locator: "github.com/torvalds/linux/blob/master/drivers/net/tun.c" } @@ -24,7 +24,7 @@ data extdeps_external_authority_anchor: ExternalAuthority = ExternalAuthority { // references held by io_uring/SCM_RIGHTS rather than installed process FDs. Denied reads refuse. // // WHERE THE QUEUE COUNT COMES FROM: the current combined channel count, read through -// extdeps.ethtool. Linux v6.9 drivers/net/tun.c tun_get_channels publishes tun->numqueues; +// extdeps.linux.ethtool_ioctl, preserving EOPNOTSUPP as a capability absence. Linux v6.9 drivers/net/tun.c tun_get_channels publishes tun->numqueues; // v6.8 has no get_channels callback. srv1's 6.8.0-138-generic returned Operation not supported // during PR #13607 commissioning on 2026-10-09. An unsupported query remains Unreadable, never // zero: descriptor sweeps alone miss SCM_RIGHTS/io_uring references. Upgrading a host requires @@ -42,6 +42,7 @@ type TunCensusRefusal | TunCensusReaderNotRoot { euid: Int } | TunCensusFlagsUnreadable | TunCensusMultiQueue + | TunCensusQueueCountUnsupported | TunCensusQueueCountUnreadable { cause: NonEmptyStr } | TunCensusProcUnreadable { path: String } @@ -52,6 +53,7 @@ fn tun_census_refusal_text(cause: TunCensusRefusal) -> String { TunCensusReaderNotRoot { euid } => concat("TUN census requires root; effective UID is ", to_string(euid)) TunCensusFlagsUnreadable => "TUN interface flags are unreadable or malformed" TunCensusMultiQueue => "TUN census does not admit multi-queue interfaces" + TunCensusQueueCountUnsupported => "TUN attached-queue count requires ETHTOOL_GCHANNELS support in the running kernel (EOPNOTSUPP)" TunCensusQueueCountUnreadable { cause: detail } => concat("TUN attached-queue count is unavailable: ", detail as String) TunCensusProcUnreadable { path } => concat("TUN descriptor census could not read ", path) } @@ -69,7 +71,7 @@ type TunDescriptorCensus fn tun_interface_name_admissible(name: String) -> Bool { let cps = name |> chars - length(cps) > 0 && all(cps, cp => (cp >= 97 && cp <= 122) || (cp >= 65 && cp <= 90) || (cp >= 48 && cp <= 57) || cp == 95 || cp == 45) + length(cps) > 0 && length(cps) < 16 && all(cps, cp => (cp >= 97 && cp <= 122) || (cp >= 65 && cp <= 90) || (cp >= 48 && cp <= 57) || cp == 95 || cp == 45) } fn tun_decimal_name(name: String) -> Bool { @@ -100,13 +102,14 @@ type TunQueueReading = TunQueuesAttached { count: Int } | TunQueuesRefused { cause: TunCensusRefusal } -fn tun_queue_reading(mode: TunQueueMode, channels: EthtoolCombinedChannels) -> TunQueueReading { +fn tun_queue_reading(mode: TunQueueMode, channels: LinuxEthtoolChannels) -> TunQueueReading { match mode { TunFlagsMalformed => TunQueuesRefused { cause: TunCensusFlagsUnreadable } TunMultiQueue => TunQueuesRefused { cause: TunCensusMultiQueue } TunSingleQueue => match channels { - EthtoolCombinedChannelsCurrent { count: n } => TunQueuesAttached { count: n } - EthtoolCombinedChannelsUnreadable { cause: c } => TunQueuesRefused { cause: TunCensusQueueCountUnreadable { cause: c } } + LinuxEthtoolChannelsRead { combined: n } => TunQueuesAttached { count: n } + LinuxEthtoolChannelsUnsupported => TunQueuesRefused { cause: TunCensusQueueCountUnsupported } + LinuxEthtoolChannelsUnreadable { cause: c } => TunQueuesRefused { cause: TunCensusQueueCountUnreadable { cause: c } } } } } @@ -233,7 +236,7 @@ fn tun_observe_queues(name: String) -> TunQueueReading { let r = Filesystem.Read(path: path) match filesystem_exact_read(path: path, content: r.content, success: r.success, error: r.error, error_kind: r.error_kind) { FilesystemExactPathRead { path: _, content: c } => - tun_queue_reading(mode: tun_queue_mode(flags: c), channels: ethtool_observe_combined_channels(device: name as NonEmptyStr)) + tun_queue_reading(mode: tun_queue_mode(flags: c), channels: linux_ethtool_channels_observe(device: name as NonEmptyStr)) other => TunQueuesRefused { cause: TunCensusFlagsUnreadable } } } @@ -250,6 +253,23 @@ fn tun_reader_refusal() -> TunCensusRefusal? { } } +// The chosen census needs this operation on this interface. Convergence consumes the same +// requirement; it does not maintain a parallel list of host kernel versions or tools. +type TunAttachedQueueRequirement { interface: NonEmptyStr } + +fn tun_attached_queue_requirement(name: NonEmptyStr) -> TunAttachedQueueRequirement { + TunAttachedQueueRequirement { interface: name } +} + +fn tun_attached_queue_requirement_observe(requirement: TunAttachedQueueRequirement) -> TunQueueReading { + if !tun_interface_name_admissible(name: requirement.interface as String) { + TunQueuesRefused { cause: TunCensusSubjectInadmissible } + } else { match tun_reader_refusal() { + Present { value: cause } => TunQueuesRefused { cause: cause } + Absent => tun_observe_queues(name: requirement.interface as String) + } } +} + // The census, in the order the refusals are owed: subject, reader identity, interface presence, // then the joined observation. Only a not_found on tun_flags is Missing; any other failure to read // it is Unreadable. @@ -263,7 +283,8 @@ fn tun_descriptor_census_observe(name: String) -> TunDescriptorCensus { match filesystem_exact_read(path: path, content: r.content, success: r.success, error: r.error, error_kind: r.error_kind) { FilesystemExactPathAbsent { path: _ } => TunInterfaceMissing FilesystemExactPathRead { path: _, content: _ } => { - let before = tun_observe_queues(name: name) + let requirement = tun_attached_queue_requirement(name: name as NonEmptyStr) + let before = tun_attached_queue_requirement_observe(requirement: requirement) match before { TunQueuesRefused { cause: c } => TunDescriptorUnreadable { cause: c } TunQueuesAttached { count: _ } => { @@ -272,7 +293,7 @@ fn tun_descriptor_census_observe(name: String) -> TunDescriptorCensus { TunSweepRefused { cause: c } => TunDescriptorUnreadable { cause: c } TunSweepComplete { holders: _, exited: _ } => { let second = tun_sweep(name: name) - tun_census_join(before: before, first: first, second: second, after: tun_observe_queues(name: name)) + tun_census_join(before: before, first: first, second: second, after: tun_attached_queue_requirement_observe(requirement: requirement)) } } } diff --git a/dag/extdeps/package_managers/apt.dag b/dag/extdeps/package_managers/apt.dag index 3c2f55ddc96..a5e7eb5ae1e 100644 --- a/dag/extdeps/package_managers/apt.dag +++ b/dag/extdeps/package_managers/apt.dag @@ -64,6 +64,22 @@ data apt_man_url: NonEmptyStr = "https://manpages.debian.org/testing/apt/apt.8.e // Install row's argv authored a second time, and its last consumer now binds `package` against // this declaration instead. service apt.PackageManager { + operation ShowExact { + input { selection: NonEmptyStr } + output { success: Bool from "exit_success", stdout: String from "stdout", stderr: String from "stderr" } + readonly + transport shell { argv: ["apt-cache", "show", "--no-all-versions", "{selection}"] } + exit { 0 => Unit nonzero => String "apt-cache could not read the selected version" } + mock_response { 1 => { success: false, stdout: "", stderr: "no package metadata supplied" } "no fabricated package candidate" } + } + operation SimulateInstall { + input { selection: NonEmptyStr } + output { success: Bool from "exit_success", stdout: String from "stdout", stderr: String from "stderr" } + readonly + transport shell { argv: ["env", "LC_ALL=C", "apt-get", "--simulate", "--no-install-recommends", "--no-remove", "install", "{selection}"] } + exit { 0 => Unit nonzero => String "apt-get simulation refused" } + mock_response { 1 => { success: false, stdout: "", stderr: "no package plan supplied" } "no fabricated install simulation" } + } operation Install { requires Network input { package: NonEmptyStr } diff --git a/dag/extdeps/package_managers/dpkg.dag b/dag/extdeps/package_managers/dpkg.dag index 8a33afa930b..969d41193a0 100644 --- a/dag/extdeps/package_managers/dpkg.dag +++ b/dag/extdeps/package_managers/dpkg.dag @@ -85,6 +85,20 @@ fn dpkg_status_lines_of(record: String) -> List { if starts_with(s: line, prefix: "Status:") { concat(acc, [trim(s: line)]) } else { acc }) } +// A status/cache stanza must name each identity field exactly once. Missing and duplicate +// fields remain unreadable; the caller checks the expected package, version and architecture. +fn dpkg_record_field(record: String, field: String) -> String? { + let prefix = concat(field, ":") + let rows = filter(split(s: record, delimiter: "\n"), line => starts_with(s: line, prefix: prefix)) + if rows.count() != 1 { none } else { match rows.first() { + Present { value: row } => { + let value = trim(s: substring(s: row, start: string_length(s: prefix), end: string_length(s: row))) + if value == "" { none } else { Present { value: value } } + } + Absent => none + } } +} + fn dpkg_words_of(text: String) -> List { fold(split(text, " "), init: [], f: (acc, w) => if w == "" { acc } else { concat(acc, [w]) }) } diff --git a/dag/extdeps/python/python.dag b/dag/extdeps/python/python.dag index e2303e5bfeb..faa303348d4 100644 --- a/dag/extdeps/python/python.dag +++ b/dag/extdeps/python/python.dag @@ -1,5 +1,7 @@ module extdeps.python +import extdeps.exec.program { ProgramIdentity, uncataloged_program } + import std.types { FilePath, List, Unit, NonEmptyStr } import std.workspace_artifact { WorkspaceFootprint, WorkspaceArtifact, CitedUpstream, RegenerableFromSource } import extdeps.external_authority { ExternalAuthority } @@ -47,6 +49,8 @@ type PythonProgramText = NonEmptyStr data python_program_flag: String = "-c" data python_interpreter_command: String = "python3" +fn python_program() -> ProgramIdentity = uncataloged_program(invocation: python_interpreter_command as NonEmptyStr) + fn python_run_program_argv(program: PythonProgramText) -> List { [python_interpreter_command, python_program_flag, program as String] } diff --git a/dag/extdeps/tools/uname.dag b/dag/extdeps/tools/uname.dag index 9c9873707cb..cacd830ecf8 100644 --- a/dag/extdeps/tools/uname.dag +++ b/dag/extdeps/tools/uname.dag @@ -35,3 +35,7 @@ fn uname_program() -> ProgramIdentity = uncataloged_program(invocation: "uname") fn uname_release_command() -> ArgvCommand { argv_command(program: uname_program(), arguments: ["-r"]) } + +fn uname_machine_command() -> ArgvCommand { + argv_command(program: uname_program(), arguments: ["-m"]) +} diff --git a/dag/gunbc/fleet/fleet_converge_plan.dag b/dag/gunbc/fleet/fleet_converge_plan.dag index d75a9ee354b..24b08d22b27 100644 --- a/dag/gunbc/fleet/fleet_converge_plan.dag +++ b/dag/gunbc/fleet/fleet_converge_plan.dag @@ -3,6 +3,9 @@ module gunbc.fleet_converge_plan import gunbc.fleet_converge_scope { FabricAllocationStoreOnly, FabricExecutionCellsOnly, FleetConvergeScope, FullHost, LaunchEnvironment, LiveDeploy, WorkspaceAllocationOnly, WorkspaceSlotCommissioningOnly } import gunbc.fleet_plan_artifact_location { fleet_converge_plan_artifact_dir } +import gunbc.host_kernel_dependency { HostKernelDependencyPlan, host_kernel_dependency_action_lines, host_kernel_maintenance_capability } +import gunbc.host_kernel_dependency_wire { host_kernel_dependency_row } + import gunbc.workspace_commissioning_fleet { WorkspaceCommissioningFleetBasis, workspace_commissioning_basis_rows } import gunbc.workspace_commissioning_executor_plan { workspace_executor_scope_wire } import gunbc.workspace_commissioning_dispatch_transport { workspace_dispatch_apply_script } @@ -407,6 +410,7 @@ type FleetConvergeRequest } | WorkspaceAllocationConverge { basis: WorkspaceConvergenceBasis, release_dir: NonEmptyStr } | WorkspaceSlotCommissioningConverge { basis: WorkspaceCommissioningFleetBasis } + | WorkspaceCommissioningDependencyConverge { plan: HostKernelDependencyPlan } | LiveDeployConverge { desired: DesiredDeployment basis: LiveDeployMemberBasis @@ -423,6 +427,7 @@ fn fleet_converge_request_scope(request: FleetConvergeRequest) -> FleetConvergeS FabricExecutionCellsConverge { host: _, observed_fabric_cells: _ } => FabricExecutionCellsOnly FabricAllocationStoreConverge { host: _, observed_allocation_store: _ } => FabricAllocationStoreOnly WorkspaceSlotCommissioningConverge { basis: _ } => WorkspaceSlotCommissioningOnly + WorkspaceCommissioningDependencyConverge { plan: _ } => WorkspaceSlotCommissioningOnly WorkspaceAllocationConverge { basis: _, release_dir: _ } => WorkspaceAllocationOnly LiveDeployConverge { desired: _, basis: _ } => LiveDeploy } @@ -435,6 +440,7 @@ fn fleet_converge_request_host(request: FleetConvergeRequest) -> NonEmptyStr { FabricExecutionCellsConverge { host: h, observed_fabric_cells: _ } => h FabricAllocationStoreConverge { host: h, observed_allocation_store: _ } => h WorkspaceSlotCommissioningConverge { basis } => basis.provenance.slot.host as NonEmptyStr + WorkspaceCommissioningDependencyConverge { plan } => plan.host as NonEmptyStr WorkspaceAllocationConverge { basis: selected, release_dir: _ } => selected.host LiveDeployConverge { desired: d, basis: _ } => d.target.host.identity as NonEmptyStr } @@ -609,6 +615,7 @@ fn fleet_converge_apply_artifact_admits( LaunchEnvironmentConverge { host: _, observed_timers: _ } => ApplyMemberSetMismatch FabricExecutionCellsConverge { host: _, observed_fabric_cells: _ } => ApplyMemberSetMismatch WorkspaceSlotCommissioningConverge { basis: _ } => ApplyMemberSetMismatch + WorkspaceCommissioningDependencyConverge { plan: _ } => ApplyMemberSetMismatch WorkspaceAllocationConverge { basis: _, release_dir: _ } => ApplyMemberSetMismatch LiveDeployConverge { desired: _, basis: _ } => ApplyMemberSetMismatch } @@ -624,6 +631,7 @@ fn fleet_converge_apply_artifact_admits( LaunchEnvironmentConverge { host: _, observed_timers: _ } => ApplyAdmitted FabricExecutionCellsConverge { host: _, observed_fabric_cells: _ } => ApplyAdmitted WorkspaceSlotCommissioningConverge { basis: _ } => ApplyAdmitted + WorkspaceCommissioningDependencyConverge { plan: _ } => ApplyAwaitingCapability { capability: host_kernel_maintenance_capability } WorkspaceAllocationConverge { basis: _, release_dir: _ } => ApplyAdmitted FabricAllocationStoreConverge { host: _, observed_allocation_store: _ } => ApplyAdmitted } @@ -1411,6 +1419,7 @@ fn fleet_converge_apply_terminal(request: FleetConvergeRequest) -> FleetConverge } } WorkspaceSlotCommissioningConverge { basis: _ } => FullyApplied + WorkspaceCommissioningDependencyConverge { plan: _ } => AwaitingCapability { capability: host_kernel_maintenance_capability } WorkspaceAllocationConverge { basis: _, release_dir: _ } => FullyApplied FabricAllocationStoreConverge { host: _, observed_allocation_store: _ } => FullyApplied LiveDeployConverge { desired: _, basis } => @@ -1458,6 +1467,7 @@ fn fleet_converge_plan_axis_lines(request: FleetConvergeRequest) -> List FabricExecutionCellsConverge { host: host, observed_fabric_cells: observed_fabric_cells } => concat(["# fabric-cell"], fleet_converge_fabric_cell_lines(host: host, observed_fabric_cells: observed_fabric_cells)) WorkspaceSlotCommissioningConverge { basis } => [workspace_commissioning_basis_rows(basis: basis)] + WorkspaceCommissioningDependencyConverge { plan } => [host_kernel_dependency_row(plan: plan)] + host_kernel_dependency_action_lines(plan: plan) WorkspaceAllocationConverge { basis: selected, release_dir: installed } => [ "# workspace allocation: dispatch existing slot controller; readiness is observed separately", concat("workspace-launch-directive:", workspace_launch_directive_text(directive: workspace_launch_directive(basis: selected, release_dir: installed))), @@ -1518,6 +1528,7 @@ fn fleet_converge_plan_refusal_counter_lines(request: FleetConvergeRequest) -> L "# allocation-store-refusals=0", ] WorkspaceSlotCommissioningConverge { basis: _ } => [concat("# scope=", workspace_executor_scope_wire)] + WorkspaceCommissioningDependencyConverge { plan: _ } => [concat("# scope=", workspace_executor_scope_wire)] WorkspaceAllocationConverge { basis: _, release_dir: _ } => [concat("# scope=", fleet_converge_scope_wire(scope: WorkspaceAllocationOnly))] LiveDeployConverge { desired: desired, basis: _ } => [ @@ -1976,6 +1987,7 @@ fn observed_baseline_text(request: FleetConvergeRequest) -> String { "\n", ) WorkspaceSlotCommissioningConverge { basis } => join([workspace_executor_scope_wire, workspace_commissioning_basis_rows(basis: basis)], "\n") + WorkspaceCommissioningDependencyConverge { plan } => join([workspace_executor_scope_wire, host_kernel_dependency_row(plan: plan)], "\n") WorkspaceAllocationConverge { basis: selected, release_dir: installed } => join( [fleet_converge_scope_wire(scope: WorkspaceAllocationOnly), concat("installed-release:", installed as String)] + workspace_convergence_baseline_rows(basis: selected), "\n", ) @@ -2535,6 +2547,7 @@ fn fleet_converge_apply_shell_axis_note(request: FleetConvergeRequest) -> String FabricAllocationStoreConverge { host: _, observed_allocation_store: _ } => "scope:fabric-allocation-store-only — allocation-store substrate ensure effects are emitted by fleet_converge_allocation_store_plan_artifact" WorkspaceSlotCommissioningConverge { basis: _ } => "commission one slot through the protected executor; wait for exact completion and durable fleet generation" + WorkspaceCommissioningDependencyConverge { plan: _ } => "commissioning requires host kernel maintenance; no slot readiness or generation may be published" WorkspaceAllocationConverge { basis: _, release_dir: _ } => "scope:workspace-allocation-only — publish typed invocation credential and start the existing selected slot controller" LiveDeployConverge { desired: _, basis: _ } => "scope:live-deploy — this script emits no host effects; wet apply is CONVERGENCE-ONE C8 and is not this arm" @@ -2584,6 +2597,7 @@ fn fleet_converge_apply_shell_axis_lines(request: FleetConvergeRequest) -> List< fabric_cell_apply_lines(host: host, observed_fabric_cells: observed_fabric_cells) FabricAllocationStoreConverge { host: _, observed_allocation_store: _ } => [] WorkspaceSlotCommissioningConverge { basis } => if basis.already_committed { [] } else { [workspace_dispatch_apply_script(release_dir: basis.release_dir, slot: runner_slot_instance_name(slot: basis.provenance.slot))] } + WorkspaceCommissioningDependencyConverge { plan: _ } => [] WorkspaceAllocationConverge { basis: selected, release_dir: installed } => if selected.allocation.phase == WorkspaceReleased { [] } else { [join(["printf '%s' ", posix_single_quote(spelling: workspace_launch_directive_text(directive: workspace_launch_directive(basis: selected, release_dir: installed))), " > ", posix_single_quote(spelling: workspace_launch_staging_path(slot: selected.reservation_plan.reservation.slot) as String)], "")] diff --git a/dag/gunbc/fleet/fleet_converge_plan_cli.dag b/dag/gunbc/fleet/fleet_converge_plan_cli.dag index e9df9ddd8d6..d3ada76e095 100644 --- a/dag/gunbc/fleet/fleet_converge_plan_cli.dag +++ b/dag/gunbc/fleet/fleet_converge_plan_cli.dag @@ -6,7 +6,10 @@ import gunbc.runner_microvm_slot_unit { microvm_controller_root } import gunbc.fleet_converge_scope { WorkspaceAllocationOnly, WorkspaceSlotCommissioningOnly, FleetConvergeScope, FullHost, FabricExecutionCellsOnly, FabricAllocationStoreOnly, LiveDeploy, LaunchEnvironment } import gunbc.fleet_plan_artifact_location { fleet_converge_plan_artifact_dir } -import gunbc.workspace_commissioning_fleet { WorkspaceCommissioningFleetObserved, WorkspaceCommissioningFleetRefused } +import gunbc.host_kernel_dependency { host_kernel_maintenance_capability } +import gunbc.host_kernel_dependency_wire { host_kernel_dependency_in_plan_body } + +import gunbc.workspace_commissioning_fleet { WorkspaceCommissioningFleetObserved, WorkspaceCommissioningFleetDependencyObserved, WorkspaceCommissioningFleetRefused } import gunbc.workspace_commissioning_dispatch_transport { workspace_dispatch_captured_command, workspace_dispatch_observe } import gunbc.workspace_commissioning_executor_plan { workspace_executor_scope_wire, workspace_executor_plan_credential, workspace_executor_plan_bytes, WorkspaceExecutorPlanBytesVerified, WorkspaceExecutorPlanBytesRefused } import gunbc.workspace_commissioning_directive { workspace_commissioning_directive_from_text, WorkspaceCommissioningDirectiveReady, WorkspaceCommissioningDirectiveRefused } @@ -180,6 +183,7 @@ import gunbc.fleet_converge_plan { FabricAllocationStoreConverge, WorkspaceAllocationConverge, WorkspaceSlotCommissioningConverge, + WorkspaceCommissioningDependencyConverge, fleet_converge_scope_of_wire, fleet_converge_scope_matches, fleet_converge_plan_artifact_of_request, @@ -880,7 +884,8 @@ fn fleet_converge_plan_scoped_bound_wet( let observed = if (scope == WorkspaceSlotCommissioningOnly) { match workspace_dispatch_observe(host: subject_host, revision: run.run_revision as CommitSha) { WorkspaceCommissioningFleetObserved { basis } => FleetConvergeRequestObserved { request: WorkspaceSlotCommissioningConverge { basis: basis } } - WorkspaceCommissioningFleetRefused { reason: _ } => FleetConvergeRequestUnobserved { reason: "commissioning privileged observation refused" } + WorkspaceCommissioningFleetDependencyObserved { plan } => FleetConvergeRequestObserved { request: WorkspaceCommissioningDependencyConverge { plan: plan } } + WorkspaceCommissioningFleetRefused { reason } => FleetConvergeRequestUnobserved { reason: concat("commissioning privileged observation refused: ", reason) } } } else if (scope == WorkspaceAllocationOnly) { observe_workspace_allocation_plan_request_wet(host: subject_host, @@ -1135,7 +1140,7 @@ fn fleet_converge_apply_ordinary_bound_wet( ApplyAllocationStorePrestateMismatch { namespace_path: n, store_path: p, observed: o } => exit_failure(reason: join(["fleet_converge_apply: AllocationStorePrestateMismatch namespace_path=", n as String, " store_path=", p as String, " observed=", o, "; rerun plan and review"], "")) ApplyAwaitingCapability { capability: c } => - exit_failure(reason: join(["fleet_converge_apply: AwaitingCapability ", c, " — nonempty live-deploy plan; wet apply is C8"], "")) + exit_failure(reason: join(["fleet_converge_apply: AwaitingCapability ", c, " — this plan requires an unavailable apply capability"], "")) ApplyAdmitted => if !fleet_converge_observed_baseline_matches( observed_baseline_hex: trim(s: read_base.content), @@ -1576,6 +1581,7 @@ fn observe_fleet_converge_request_from_plan_wet(scope_wire: String, host: NonEmp Present { value: credential } => match workspace_commissioning_directive_from_text(text: credential) { WorkspaceCommissioningDirectiveReady { provenance } => match workspace_dispatch_observe(host: host, revision: provenance.source_revision as CommitSha) { WorkspaceCommissioningFleetObserved { basis } => FleetConvergeRequestObserved { request: WorkspaceSlotCommissioningConverge { basis: basis } } + WorkspaceCommissioningFleetDependencyObserved { plan: _ } => FleetConvergeRequestUnobserved { reason: "commissioning dependency remains unresolved" } WorkspaceCommissioningFleetRefused { reason: _ } => FleetConvergeRequestUnobserved { reason: "commissioning no-op readback refused" } } WorkspaceCommissioningDirectiveRefused { reason: _ } => FleetConvergeRequestUnobserved { reason: "commissioning provenance unreadable" } @@ -1625,7 +1631,9 @@ fn fleet_converge_apply_bound_wet(run: ActionsRunBinding, plan_run_id: String, plan_receipt: FleetConvergePlanReceipt, expected_revision_admitted: String) -> ProcessExit { let body = Filesystem.Read(path: fleet_converge_plan_body_path) let noop = body.success && count(filter(split(s: body.content, delimiter: "\n"), row => row == "commissioning-already-committed:true")) == 1 - if plan_receipt.scope_wire != workspace_executor_scope_wire || noop { + if plan_receipt.scope_wire == workspace_executor_scope_wire && body.success && host_kernel_dependency_in_plan_body(body: body.content) { + exit_failure(reason: concat("fleet_converge_apply: AwaitingCapability — ", host_kernel_maintenance_capability)) + } else if plan_receipt.scope_wire != workspace_executor_scope_wire || noop { fleet_converge_apply_ordinary_bound_wet(run: run, plan_run_id: plan_run_id, plan_receipt: plan_receipt, expected_revision_admitted: expected_revision_admitted) } else { fleet_converge_apply_commissioning_bound_wet(run: run, plan_run_id: plan_run_id, diff --git a/dag/gunbc/fleet/fleet_converge_workflow.dag b/dag/gunbc/fleet/fleet_converge_workflow.dag index 7d6837d971c..a7fdb453f2a 100644 --- a/dag/gunbc/fleet/fleet_converge_workflow.dag +++ b/dag/gunbc/fleet/fleet_converge_workflow.dag @@ -4449,12 +4449,7 @@ fn fleet_converge_credential_prelude_steps() -> List { } fn fleet_converge_job() -> Job { - Job { - id: fleet_converge_shared_job_id as String, - name: none, - runner: fleet_converge_host_pinned_runner_spec(), - environment: none, - steps: concat( + fleet_converge_job_with_steps(steps: concat( concat( fleet_converge_consumer_prelude_steps(), fleet_converge_credential_prelude_steps() @@ -4575,7 +4570,17 @@ fn fleet_converge_job() -> Job { fleet_converge_site_pxe_edge_receipt_upload_step(), ci_fleet_key_agent_cleanup_step(), ] - ), + )) +} + +// The shared job envelope owns selection and credentials independently of its step producers. +fn fleet_converge_job_with_steps(steps: List) -> Job { + Job { + id: fleet_converge_shared_job_id as String, + name: none, + runner: fleet_converge_host_pinned_runner_spec(), + environment: none, + steps: steps, needs: fleet_converge_job_needs(id: fleet_converge_shared_job_id), env: none, outputs: none, diff --git a/dag/gunbc/host/host_kernel_dependency.dag b/dag/gunbc/host/host_kernel_dependency.dag new file mode 100644 index 00000000000..f33539b7579 --- /dev/null +++ b/dag/gunbc/host/host_kernel_dependency.dag @@ -0,0 +1,157 @@ +module gunbc.host_kernel_dependency + +import std.types { String, NonEmptyStr, Bool, List, CommitSha } +import std.algebra { trim } +import std.upsert_decision { Noop, Apply, Refuse, UpsertDecision } +import gunbc.ensure { ensure_decide, EnsureMayApply, EnsureObserved, EnsureInaccessible, EnsurePlanAvailable, EnsurePlanUnavailable } +import extdeps.linux.kernel { LinuxKernelRelease } +import extdeps.linux.tun_fdinfo { TunAttachedQueueRequirement, TunQueueReading, TunQueuesAttached, TunQueuesRefused, TunCensusQueueCountUnsupported, tun_census_refusal_text } +import extdeps.dpkg { dpkg_record_field, dpkg_query_status_verdict_of, DpkgPackageInstalled, DpkgPackageNotInstalled, DpkgPackageUnknown, DpkgQueryUndecidable } +import extdeps.shell.exec { ProcessObservation } +import product.host_identity { HostIdentity } +import gunbc.fleet_host_identity { operator_host_srv1 } + +// A provider selection is host policy, never a requirement authored by VM code. The selected +// TUN census asks for TunAttachedQueueRequirement. This Ubuntu realization can establish it +// by installing and booting a kernel, but only the actual queue query establishes satisfaction. +// No minimum-version comparison: a backport that answers the query needs no mutation. +type UbuntuTunKernelProvider { + package: NonEmptyStr + version: NonEmptyStr + architecture: NonEmptyStr + kernel_release: LinuxKernelRelease +} + +// Measured package candidate and downloaded System.map (tun_get_channels), 2026-10-09. +// Selection is permission to propose this realization, not authorization to interrupt a host. +fn tun_kernel_provider_for_host(host: HostIdentity) -> UbuntuTunKernelProvider? { + if host == operator_host_srv1 { Present { value: UbuntuTunKernelProvider { + package: "linux-generic-hwe-24.04", version: "7.0.0-38.38~24.04.4", + architecture: "arm64", kernel_release: "7.0.0-38-generic" as LinuxKernelRelease, + } } } else { none } +} + +fn kernel_provider_package_names(provider: UbuntuTunKernelProvider) -> List { + [provider.package, concat("linux-image-", provider.kernel_release as String) as NonEmptyStr, + concat("linux-modules-", provider.kernel_release as String) as NonEmptyStr] +} + +fn kernel_provider_selection(provider: UbuntuTunKernelProvider) -> NonEmptyStr { + concat(provider.package as String, concat("=", provider.version as String)) as NonEmptyStr +} + +type KernelPackageReading + = KernelPackageExact + | KernelPackageNeedsInstall + | KernelPackageReadRefused { reason: String } + +fn kernel_package_reading(package: NonEmptyStr, provider: UbuntuTunKernelProvider, reading: ProcessObservation) -> KernelPackageReading { + match dpkg_query_status_verdict_of(exit_code: reading.exit_code, stdout: reading.stdout, stderr: reading.stderr) { + DpkgQueryUndecidable { reason } => KernelPackageReadRefused { reason: reason } + DpkgPackageUnknown => KernelPackageNeedsInstall + DpkgPackageNotInstalled { status } => if starts_with(s: status, prefix: "Status: hold ") { + KernelPackageReadRefused { reason: "selected kernel package is held and not installed" } + } else { KernelPackageNeedsInstall } + DpkgPackageInstalled => { + let name = dpkg_record_field(record: reading.stdout, field: "Package") + let version = dpkg_record_field(record: reading.stdout, field: "Version") + let arch = dpkg_record_field(record: reading.stdout, field: "Architecture") + if name != Present { value: package as String } || arch != Present { value: provider.architecture as String } + || version == none { + KernelPackageReadRefused { reason: "kernel package identity or architecture is unreadable or different" } + } else if version == Present { value: provider.version as String } { KernelPackageExact } + else { KernelPackageReadRefused { reason: "an installed kernel package differs from the selected version; refuse implicit downgrade or held-package replacement" } } + } + } +} + +fn kernel_package_readings_join(provider: UbuntuTunKernelProvider, readings: List) -> KernelPackageReading { + if readings.count() != kernel_provider_package_names(provider: provider).count() { KernelPackageReadRefused { reason: "kernel provider package census is incomplete" } } + else { fold(readings, init: KernelPackageExact, f: (acc, item) => match acc { + KernelPackageReadRefused { reason: _ } => acc + other => match item { + KernelPackageReadRefused { reason: _ } => item + KernelPackageNeedsInstall => KernelPackageNeedsInstall + KernelPackageExact => acc + } + }) } +} + +type HostKernelRemedy + = InstallKernelAndActivate + | ActivateInstalledKernel + +// Intention only. Neither this plan nor its package simulation may mint cell readiness. +// The apply effect is held at the existing AwaitingCapability boundary until host maintenance +// authority, drained work and an external reset observer have been established for THIS plan. +type HostKernelDependencyPlan { + host: HostIdentity + source_revision: CommitSha + requirement: TunAttachedQueueRequirement + running_release: LinuxKernelRelease + boot_id: NonEmptyStr + provider: UbuntuTunKernelProvider + remedy: HostKernelRemedy + package_simulation: String +} + +type HostKernelDependencyResolution + = HostKernelCapabilitySatisfied + | HostKernelDependencyPlanned { plan: HostKernelDependencyPlan } + | HostKernelDependencyRefused { reason: String } + +// An operation's observed support decides whether any provider is demanded. A positive count +// means the query is supported, NOT that sanitation is clean. The unchanged census still +// rejects held devices. Only EOPNOTSUPP, established by the ioctl, reaches provider selection. +fn tun_kernel_capability_decision(requirement: TunAttachedQueueRequirement, reading: TunQueueReading) -> UpsertDecision { + match reading { + TunQueuesAttached { count: _ } => ensure_decide(policy: EnsureMayApply, + observation: EnsureObserved { evidence: reading }, observation_satisfies: true, + mismatch_reason: "", plan: EnsurePlanUnavailable { reason: "no remedy is demanded" }) + TunQueuesRefused { cause: TunCensusQueueCountUnsupported } => ensure_decide(policy: EnsureMayApply, + observation: EnsureObserved { evidence: reading }, observation_satisfies: false, + mismatch_reason: "running kernel does not support the selected TUN queue query", + plan: EnsurePlanAvailable { plan: requirement }) + TunQueuesRefused { cause } => ensure_decide(policy: EnsureMayApply, + observation: EnsureInaccessible { reason: tun_census_refusal_text(cause: cause) }, observation_satisfies: false, + mismatch_reason: "", plan: EnsurePlanUnavailable { reason: "unreadable is not absent" }) + } +} + +fn host_kernel_dependency_plan(host: HostIdentity, revision: CommitSha, requirement: TunAttachedQueueRequirement, + running_release: LinuxKernelRelease, boot_id: NonEmptyStr, provider: UbuntuTunKernelProvider, + packages: KernelPackageReading, simulation: String) -> HostKernelDependencyResolution { + if running_release == provider.kernel_release { + HostKernelDependencyRefused { reason: "the selected kernel is already running but the required query is unsupported; refuse a reboot loop" } + } else if (packages == KernelPackageNeedsInstall && trim(s: simulation) == "") + || (packages == KernelPackageExact && simulation != "") { + HostKernelDependencyRefused { reason: "kernel dependency plan lacks the simulation appropriate to its package state" } + } else { match packages { + KernelPackageReadRefused { reason } => HostKernelDependencyRefused { reason: reason } + other => HostKernelDependencyPlanned { plan: HostKernelDependencyPlan { + host: host, source_revision: revision, requirement: requirement, running_release: running_release, + boot_id: boot_id, provider: provider, + remedy: match packages { KernelPackageExact => ActivateInstalledKernel other => InstallKernelAndActivate }, + package_simulation: simulation, + } } + } } +} + +fn host_kernel_dependency_action_lines(plan: HostKernelDependencyPlan) -> List { + let install = match plan.remedy { + InstallKernelAndActivate => [concat("install-package:", kernel_provider_selection(provider: plan.provider) as String)] + ActivateInstalledKernel => [] + } + ["requirement:safe-slot-reuse -> TUN descriptor census -> ETHTOOL_GCHANNELS combined_count", + concat("required-interface:", plan.requirement.interface as String), + concat("observed-running-kernel:", plan.running_release as String), + "observed-capability:unsupported (errno EOPNOTSUPP)"] + install + [ + concat("activate-kernel:", plan.provider.kernel_release as String), + "activation-prerequisites: plan-bound host interruption authority; drained runners; verified console recovery; external reset observer", + "readback: running kernel and new boot identity; KVM; network and services; successful TUN queue query", + "continuation: rerun the same commissioning goal; sanitation must still prove zero holders", + "standing: dependency remedy planned; VM remains uncommissioned", + ] +} + +data host_kernel_maintenance_capability: String = "plan-bound host kernel maintenance: interruption authority, runner drain, console recovery and external boot observation" diff --git a/dag/gunbc/host/host_kernel_dependency_observe.dag b/dag/gunbc/host/host_kernel_dependency_observe.dag new file mode 100644 index 00000000000..48a7fadd0d2 --- /dev/null +++ b/dag/gunbc/host/host_kernel_dependency_observe.dag @@ -0,0 +1,125 @@ +module gunbc.host_kernel_dependency_observe + +import std.types { String, NonEmptyStr, CommitSha, Bool } +import std.algebra { trim } +import std.upsert_decision { Noop, Apply, Refuse } +import extdeps.linux.kernel { LinuxKernelRelease } +import extdeps.linux.tun_fdinfo { TunAttachedQueueRequirement, tun_attached_queue_requirement_observe } +import extdeps.filesystem.filesystem_io { Filesystem } +import extdeps.tools.hostname { hostname_short_read_command } +import extdeps.tools.uname { uname_release_command, uname_machine_command } +import extdeps.dpkg { dpkg_query_status_command, dpkg_record_field } +import extdeps.apt +import extdeps.exec.command { ArgvCommand, LocalExec } +import extdeps.shell.exec { ProcessRan, ProcessNotSpawned } +import gunbc.command_runner { run_shell_command_observe_outcome } +import product.host_identity { HostIdentity } +import gunbc.host_kernel_dependency { + UbuntuTunKernelProvider, tun_kernel_provider_for_host, kernel_provider_package_names, kernel_provider_selection, + KernelPackageReading, KernelPackageExact, KernelPackageNeedsInstall, KernelPackageReadRefused, + kernel_package_reading, kernel_package_readings_join, tun_kernel_capability_decision, host_kernel_dependency_plan, + HostKernelDependencyResolution, HostKernelCapabilitySatisfied, HostKernelDependencyPlanned, HostKernelDependencyRefused, +} + +fn kernel_read_command(command: ArgvCommand) -> String? { + match run_shell_command_observe_outcome(command: command, transport: LocalExec) { + ProcessRan { observation } => if observation.exit_code == 0 && trim(s: observation.stdout) != "" { + Present { value: trim(s: observation.stdout) } + } else { none } + ProcessNotSpawned { program: _, cause: _ } => none + } +} + +fn kernel_provider_package_observe(package: NonEmptyStr, provider: UbuntuTunKernelProvider) -> KernelPackageReading { + match run_shell_command_observe_outcome(command: dpkg_query_status_command(package: package), transport: LocalExec) { + ProcessRan { observation } => kernel_package_reading(package: package, provider: provider, reading: observation) + ProcessNotSpawned { program: _, cause } => KernelPackageReadRefused { reason: cause as String } + } +} + +fn kernel_provider_platform_admits(os_release: String, machine: String, provider: UbuntuTunKernelProvider) -> Bool { + let lines = split(s: os_release, delimiter: "\n") + filter(lines, line => starts_with(s: line, prefix: "ID=")).count() == 1 + && filter(lines, line => starts_with(s: line, prefix: "VERSION_ID=")).count() == 1 + && any(lines, line => line == "ID=ubuntu" || line == "ID=\"ubuntu\"") + && any(lines, line => line == "VERSION_ID=24.04" || line == "VERSION_ID=\"24.04\"") + && machine == "aarch64" && provider.architecture == "arm64" +} + +fn kernel_provider_candidate_admits(provider: UbuntuTunKernelProvider, record: String) -> Bool { + dpkg_record_field(record: record, field: "Package") == Present { value: provider.package as String } + && dpkg_record_field(record: record, field: "Version") == Present { value: provider.version as String } + && dpkg_record_field(record: record, field: "Architecture") == Present { value: provider.architecture as String } +} + +// The ordinary capability dependency is evaluated before provider lookup or any package read. +// This makes an already-capable host a no-op even when it uses a different package or a backport. +fn host_tun_kernel_dependency_observe(host: HostIdentity, revision: CommitSha, + requirement: TunAttachedQueueRequirement) -> HostKernelDependencyResolution { + if kernel_read_command(command: hostname_short_read_command()) != Present { value: host as String } { + HostKernelDependencyRefused { reason: "kernel dependency must be observed on its bound host" } + } else { match tun_kernel_capability_decision(requirement: requirement, reading: tun_attached_queue_requirement_observe(requirement: requirement)) { + Noop => HostKernelCapabilitySatisfied + Refuse { reason } => HostKernelDependencyRefused { reason: reason as String } + Apply { plan: demanded } => match tun_kernel_provider_for_host(host: host) { + Absent => HostKernelDependencyRefused { reason: "the selected TUN census requires kernel channel support but this host has no admitted provider" } + Present { value: provider } => host_tun_kernel_provider_observe(host: host, revision: revision, + requirement: demanded, provider: provider) + } + } } +} + +fn host_tun_kernel_provider_observe(host: HostIdentity, revision: CommitSha, + requirement: TunAttachedQueueRequirement, provider: UbuntuTunKernelProvider) -> HostKernelDependencyResolution { + let os = Filesystem.Read(path: "/etc/os-release") + let boot = Filesystem.Read(path: "/proc/sys/kernel/random/boot_id") + let machine = kernel_read_command(command: uname_machine_command()) + let running = kernel_read_command(command: uname_release_command()) + if !os.success || !boot.success || trim(s: boot.content) == "" || machine == none || running == none { + HostKernelDependencyRefused { reason: "kernel provider platform or running boot identity is unreadable" } + } else { match machine { + Absent => HostKernelDependencyRefused { reason: "kernel provider machine architecture is unreadable" } + Present { value: arch } => if !kernel_provider_platform_admits(os_release: os.content, machine: arch, provider: provider) { + HostKernelDependencyRefused { reason: "kernel provider is not admitted for the observed distribution and architecture" } + } else { match running { + Absent => HostKernelDependencyRefused { reason: "running kernel release is unreadable" } + Present { value: kernel } => { + let packages = kernel_package_readings_join(provider: provider, + readings: map(kernel_provider_package_names(provider: provider), name => kernel_provider_package_observe(package: name, provider: provider))) + let decision = match packages { + KernelPackageReadRefused { reason } => HostKernelDependencyRefused { reason: reason } + KernelPackageNeedsInstall => { + let candidate = apt.PackageManager.ShowExact(selection: kernel_provider_selection(provider: provider)) + if !candidate.success || !kernel_provider_candidate_admits(provider: provider, record: candidate.stdout) { + HostKernelDependencyRefused { reason: "the approved kernel package version is unavailable or has different metadata" } + } else { + let simulation = apt.PackageManager.SimulateInstall(selection: kernel_provider_selection(provider: provider)) + if !simulation.success || trim(s: simulation.stdout) == "" { + HostKernelDependencyRefused { reason: "the selected kernel package has no successful non-removing installation plan" } + } else { host_kernel_dependency_plan(host: host, revision: revision, requirement: requirement, + running_release: kernel as LinuxKernelRelease, boot_id: trim(s: boot.content) as NonEmptyStr, + provider: provider, packages: packages, simulation: simulation.stdout) } + } + } + KernelPackageExact => { + let entries = Filesystem.List(path: "/boot") + let symbols = Filesystem.Read(path: concat("/boot/System.map-", provider.kernel_release as String)) + let names = split(s: entries.entries, delimiter: "\n") + if !entries.success || !symbols.success + || !any(names, name => name == concat("vmlinuz-", provider.kernel_release as String)) + || !any(names, name => name == concat("initrd.img-", provider.kernel_release as String)) + || !any(split(s: symbols.content, delimiter: "\n"), line => line.ends_with(suffix: " tun_get_channels")) { + HostKernelDependencyRefused { reason: "installed kernel package lacks a verified boot image, initramfs or TUN channel implementation" } + } else { host_kernel_dependency_plan(host: host, revision: revision, requirement: requirement, + running_release: kernel as LinuxKernelRelease, boot_id: trim(s: boot.content) as NonEmptyStr, + provider: provider, packages: packages, simulation: "") } + } + } + let after = Filesystem.Read(path: "/proc/sys/kernel/random/boot_id") + if !after.success || after.content != boot.content { + HostKernelDependencyRefused { reason: "host boot changed while observing the kernel dependency" } + } else { decision } + } + } } + } } +} diff --git a/dag/gunbc/host/host_kernel_dependency_wire.dag b/dag/gunbc/host/host_kernel_dependency_wire.dag new file mode 100644 index 00000000000..4f3ca8e3808 --- /dev/null +++ b/dag/gunbc/host/host_kernel_dependency_wire.dag @@ -0,0 +1,81 @@ +module gunbc.host_kernel_dependency_wire + +import std.types { String, NonEmptyStr, CommitSha, Bool } +import extdeps.languages.json.emit { JsonValue, json_object, json_kv, json_string, serialize_json } +import extdeps.languages.json.parse { parse_json_document, JsonDocumentParsed, JsonDocumentUnreadable, json_field_string, FieldRead, FieldAbsent, FieldMalformed } +import extdeps.linux.kernel { LinuxKernelRelease } +import extdeps.linux.tun_fdinfo { TunAttachedQueueRequirement } +import product.host_identity { HostIdentity } +import gunbc.host_kernel_dependency { HostKernelDependencyPlan, InstallKernelAndActivate, ActivateInstalledKernel, tun_kernel_provider_for_host, kernel_provider_selection } + +data host_kernel_dependency_row_prefix: String = "commissioning-kernel-dependency:" + +fn host_kernel_dependency_text(plan: HostKernelDependencyPlan) -> String { + serialize_json(v: json_object(members: [ + json_kv(key: "schema", value: json_string(s: "host-kernel-dependency/v1")), + json_kv(key: "host", value: json_string(s: plan.host as String)), + json_kv(key: "source_revision", value: json_string(s: plan.source_revision as String)), + json_kv(key: "operation", value: json_string(s: "linux.tun.ETHTOOL_GCHANNELS")), + json_kv(key: "interface", value: json_string(s: plan.requirement.interface as String)), + json_kv(key: "running_release", value: json_string(s: plan.running_release as String)), + json_kv(key: "boot_id", value: json_string(s: plan.boot_id as String)), + json_kv(key: "package", value: json_string(s: kernel_provider_selection(provider: plan.provider) as String)), + json_kv(key: "architecture", value: json_string(s: plan.provider.architecture as String)), + json_kv(key: "target_release", value: json_string(s: plan.provider.kernel_release as String)), + json_kv(key: "remedy", value: json_string(s: match plan.remedy { InstallKernelAndActivate => "install-and-activate" ActivateInstalledKernel => "activate-installed" })), + json_kv(key: "package_simulation", value: json_string(s: plan.package_simulation)), + ])) +} + +fn host_kernel_dependency_row(plan: HostKernelDependencyPlan) -> String { + concat(host_kernel_dependency_row_prefix, host_kernel_dependency_text(plan: plan)) +} + +fn kernel_dependency_field(doc: JsonValue, name: String) -> String? { + match json_field_string(obj: doc, key: name) { + FieldRead { value } => Present { value: value } + FieldAbsent => none + FieldMalformed { cause: _ } => none + } +} + +// Only the privileged observer's captured response uses this decoder. Reconstruct the policy +// selection from the host, then require canonical equality: fields cannot select another package, +// requirement, host or source revision. A plan is still intention, never an authorization token. +fn host_kernel_dependency_from_text(text: String, host: HostIdentity, revision: CommitSha, + requirement: TunAttachedQueueRequirement) -> HostKernelDependencyPlan? { + match parse_json_document(s: text) { + JsonDocumentUnreadable { gap: _ } => none + JsonDocumentParsed { value: doc } => match tun_kernel_provider_for_host(host: host) { + Absent => none + Present { value: provider } => match kernel_dependency_field(doc: doc, name: "running_release") { + Absent => none + Present { value: running } => match kernel_dependency_field(doc: doc, name: "boot_id") { + Absent => none + Present { value: boot } => match kernel_dependency_field(doc: doc, name: "remedy") { + Absent => none + Present { value: remedy } => match kernel_dependency_field(doc: doc, name: "package_simulation") { + Absent => none + Present { value: simulation } => if running == "" || boot == "" || running == (provider.kernel_release as String) + || (remedy != "install-and-activate" && remedy != "activate-installed") + || (remedy == "install-and-activate" && simulation == "") + || (remedy == "activate-installed" && simulation != "") { none } + else { + let plan = HostKernelDependencyPlan { host: host, source_revision: revision, + requirement: requirement, running_release: running as LinuxKernelRelease, boot_id: boot as NonEmptyStr, + provider: provider, remedy: if remedy == "install-and-activate" { InstallKernelAndActivate } else { ActivateInstalledKernel }, + package_simulation: simulation } + if host_kernel_dependency_text(plan: plan) == text { Present { value: plan } } else { none } + } + } + } + } + } + } + } +} + +// This marker can only stop apply; it never admits an action or decodes a readiness permit. +fn host_kernel_dependency_in_plan_body(body: String) -> Bool { + any(split(s: body, delimiter: "\n"), row => starts_with(s: row, prefix: host_kernel_dependency_row_prefix)) +} diff --git a/dag/gunbc/recurring_failure_mode/optional_equality_answers_by_representation.dag b/dag/gunbc/recurring_failure_mode/optional_equality_answers_by_representation.dag index dccb6715809..005b6c7cf35 100644 --- a/dag/gunbc/recurring_failure_mode/optional_equality_answers_by_representation.dag +++ b/dag/gunbc/recurring_failure_mode/optional_equality_answers_by_representation.dag @@ -16,6 +16,8 @@ data optional_equality_answers_by_representation: RecurringFailureMode = Recurri "EVIDENCE BY EXECUTION: test.claim.optional_equality_by_representation. RED a_present_first_equals_its_present_literal is `.first()` over a data list of variant rows, compared with a Present literal. It is enrolled in v2.workflow.floor_expected_red, so it executes on every floor and is held as known-red. POSITIVE CONTROL a_match_on_the_present_first_reads_its_value is the same read eliminated by a match, an ordinary claim. First observed as probes zz_a1 (FAIL) and zz_c1 (PASS) over test.claim.os_install_actuator_selection at gunbc#13536's base.", + "2026-10-09, PR #13607: the ETHTOOL_GCHANNELS reply decoder compared numbers.first() against Present containing the command string. The live ABI probe returned the correct nine u32s, and all numeric checks passed, but the valid-reply witness failed because this comparison answered false for 60 against Present 60. The decoder follows the existing match discipline; its positive count, wrong-command, malformed and foreign-interface controls remain enrolled. This repairs that consumer, not the seed's known representation divergence.", + "RUNG FOUND AT: below the ladder on the interpretation path. CEILING: 4. NEXT-RUNG TRIGGER, NAMING THE CAPABILITY: one authority for `first` from which the interpreter and the emitted realization are both derived, so the interpreter's `first` yields the modeled optional on every route (method form and pipe form). When it lands, both enrolled reds (this one and interpreter_first_returns_the_modeled_optional) pass, their roster rows are deleted, and the claims remain as regression controls (DESIGN 4b(4)). Until then, the interim discipline at a `first` read is to match on the optional.", ], diff --git a/dag/gunbc/recurring_failure_mode/unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider.dag b/dag/gunbc/recurring_failure_mode/unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider.dag new file mode 100644 index 00000000000..58d3cf5170a --- /dev/null +++ b/dag/gunbc/recurring_failure_mode/unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider.dag @@ -0,0 +1,14 @@ +module gunbc.recurring_failure_mode.unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider + +import std.types { NonEmptyStr } +import gunbc.recurring_failure_mode { RecurringFailureMode } + +data unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider: RecurringFailureMode = RecurringFailureMode { + identity: "unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider" as NonEmptyStr, + receipts: [ + "2026-10-09, PR #13607: safe workspace reuse selected a TUN descriptor census requiring the kernel attached-queue query, but commissioning carried only its failure text. The earliest missing edge was from the selected operation to its environmental requirement, before a kernel package or reboot could be chosen. The CLI reader also collapsed unsupported and unreadable into one refusal, so it could not safely select a remedy.", + "The census now owns TunAttachedQueueRequirement. The read-only Linux ioctl reader preserves numeric EOPNOTSUPP; all other failures stay unreadable. Commissioning consumes that requirement and the existing ensure decision: a supported query needs no provider, including a positive queue count that still fails sanitation; unsupported reaches host-selected Ubuntu package/activation planning. Host/revision/interface/boot and package observations bind the fleet plan. No version comparison establishes capability and the same already-running unsupported target refuses a reboot loop.", + "The production commissioning scope carries the derived dependency and the existing AwaitingCapability terminal. srv1 has no admitted reset selection, verified runner drain or console recovery, and this session has no shared-host reboot approval. Those are an explicit maintenance-actuation frontier, not a successful apply. Generic compiler completion described in ensure-closure-design remains unimplemented. Attainable ceiling: the operation demand is shared by construction; external running-kernel support requires live readback. The package/read-error controls and fleet artifact/admission witness exercise production folds; the wet reader witness executes the Python/native ioctl boundary.", + ], + evidence: [], +} diff --git a/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag b/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag index 6fdea2b8ae3..0ef4fe1b202 100644 --- a/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag +++ b/dag/gunbc/runner/runner_microvm_network_sanitation_observe.dag @@ -4,7 +4,8 @@ import std.types { String, NonEmptyStr, Bool } import std.algebra { trim } import std.resources { Filesystem } import std.content_hash { sha256_hex_digest } -import extdeps.linux.tun_fdinfo { tun_descriptor_census_observe, tun_census_refusal_text, TunDescriptorCensus, TunDescriptorCount, TunInterfaceMissing, TunDescriptorUnreadable } +import extdeps.linux.tun_fdinfo { tun_descriptor_census_observe, tun_census_refusal_text, TunDescriptorCensus, TunDescriptorCount, TunInterfaceMissing, TunDescriptorUnreadable, TunAttachedQueueRequirement, tun_attached_queue_requirement } +import gunbc.build_cache_instance { RunnerSlotIdentity } import extdeps.exec.command { ArgvCommand, LocalExec } import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent } import extdeps.tools.sha256sum { sha256sum_stdin_digest_via_shell, Sha256FileDigest } @@ -21,6 +22,12 @@ import gunbc.runner_microvm_network { runner_slot_guest_neighbour_show_command, runner_host_expectation_list_command, runner_microvm_nft_list_command, } +// The selected sanitation reader owns its environmental demand. Commissioning consumes this +// same requirement; host policy chooses a provider only if this operation is unsupported. +fn slot_network_tun_queue_requirement(slot: RunnerSlotIdentity) -> TunAttachedQueueRequirement { + tun_attached_queue_requirement(name: runner_slot_network(slot: slot).tap_device) +} + fn observe_tap_holders(tap: NonEmptyStr) -> TapHolderObservation { tap_holders_reading(census: tun_descriptor_census_observe(name: tap as String)) } @@ -65,7 +72,7 @@ fn observe_slot_network_readings(subject: SlotNetworkSanitationSubject) -> SlotN else if trim(s: ipv6.content) == "0" { TapIpv6Enabled } else { TapIpv6Unread { reason: "tap IPv6 policy malformed" as NonEmptyStr } } SlotNetworkReadings { - tap: TapReading { subject: subject.subject, observation: observe_tap_holders(tap: net.tap_device) }, + tap: TapReading { subject: subject.subject, observation: observe_tap_holders(tap: slot_network_tun_queue_requirement(slot: subject.subject.slot).interface) }, tap_ipv6: TapIpv6Reading { subject: subject.subject, observation: ipv6_standing }, guest_originated_conntrack: SlotResidueReading { subject: subject.subject, observation: observe_network_residue(command: runner_slot_guest_originated_conntrack_list_command(net: net)) }, guest_destined_conntrack: SlotResidueReading { subject: subject.subject, observation: observe_network_residue(command: runner_slot_guest_destined_conntrack_list_command(net: net)) }, diff --git a/dag/gunbc/workspace_commissioning_dispatch.dag b/dag/gunbc/workspace_commissioning_dispatch.dag index 34abb8e9359..ad903fd1d8e 100644 --- a/dag/gunbc/workspace_commissioning_dispatch.dag +++ b/dag/gunbc/workspace_commissioning_dispatch.dag @@ -21,7 +21,8 @@ import gunbc.workspace_commissioning_executor_input { workspace_executor_input_f import gunbc.workspace_commissioning_executor_unit { workspace_executor_unit_instance_name } import gunbc.workspace_commissioning_executor_store { workspace_executor_journal_observe, WorkspaceExecutorJournalObserved, WorkspaceExecutorJournalUnreadable } import gunbc.workspace_commissioning_executor_journal { ExecutorGenerationCommitted } -import gunbc.workspace_commissioning_fleet { workspace_commissioning_basis_observe, workspace_commissioning_candidate, workspace_commissioning_basis_rows, WorkspaceCommissioningFleetObserved, WorkspaceCommissioningFleetRefused } +import gunbc.host_kernel_dependency_wire { host_kernel_dependency_row } +import gunbc.workspace_commissioning_fleet { workspace_commissioning_basis_observe, workspace_commissioning_candidate, workspace_commissioning_basis_rows, WorkspaceCommissioningFleetObserved, WorkspaceCommissioningFleetDependencyObserved, WorkspaceCommissioningFleetRefused } import gunbc.workspace_commissioning_start_observe { workspace_commissioning_start_observe, WorkspaceCommissioningStartObserved, WorkspaceCommissioningStartSnapshot, workspace_commissioning_start_quiescent, WorkspaceCommissioningStartUnreadable } import gunbc.build_cache_instance { runner_slot_instance_name } import gunbc.fleet_generation_execution { fleet_generation_publication_as_owner } @@ -65,6 +66,7 @@ fn workspace_dispatch_stage(bytes: WorkspaceCommissioningInputBytes) -> CliWireR WorkspaceCommissioningInputPresent { bytes: stored, plan: _ } => stored == bytes WorkspaceCommissioningInputAbsent => match workspace_commissioning_basis_observe(provenance: plan.provenance) { WorkspaceCommissioningFleetObserved { basis } => !basis.already_committed && basis.prestate == plan.prestate + WorkspaceCommissioningFleetDependencyObserved { plan: _ } => false WorkspaceCommissioningFleetRefused { reason: _ } => false } WorkspaceCommissioningInputRefused { reason: _ } => false @@ -180,6 +182,7 @@ fn workspace_dispatch_plan_cli() -> CliWireResponse { workspace_dispatch_refused(reason: "plan revision is malformed") } else { match workspace_commissioning_candidate(host: host as NonEmptyStr, revision: revision as CommitSha) { WorkspaceCommissioningFleetObserved { basis } => CliWirePrintable { bytes: workspace_commissioning_basis_rows(basis: basis), exit: ExitSuccess } + WorkspaceCommissioningFleetDependencyObserved { plan } => CliWirePrintable { bytes: host_kernel_dependency_row(plan: plan), exit: ExitSuccess } WorkspaceCommissioningFleetRefused { reason } => workspace_dispatch_refused(reason: concat("privileged commissioning observation refused: ", reason)) } } } diff --git a/dag/gunbc/workspace_commissioning_executor_submission.dag b/dag/gunbc/workspace_commissioning_executor_submission.dag index ecb46d8ab9b..0e06b5d6e53 100644 --- a/dag/gunbc/workspace_commissioning_executor_submission.dag +++ b/dag/gunbc/workspace_commissioning_executor_submission.dag @@ -44,7 +44,7 @@ import gunbc.command_runner { run_shell_command_capture } import extdeps.exec.command { LocalExec } import extdeps.shell.exec { ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } import std.algebra { trim } -import gunbc.workspace_commissioning_fleet { workspace_commissioning_basis_observe, WorkspaceCommissioningFleetObserved, WorkspaceCommissioningFleetRefused } +import gunbc.workspace_commissioning_fleet { workspace_commissioning_basis_observe, WorkspaceCommissioningFleetObserved, WorkspaceCommissioningFleetDependencyObserved, WorkspaceCommissioningFleetRefused } type WorkspaceExecutorPreparation = WorkspaceExecutorPrepared { directive: String } @@ -203,6 +203,7 @@ fn workspace_executor_prepare(slot: NonEmptyStr, executor: NonEmptyStr) -> Works Present { value: generation } => workspace_executor_arm(input: input, executor: executor, snapshot: snapshot, generation: generation) Absent => WorkspaceExecutorPreparationRefused { reason: "initial mutation generation is unavailable" } } } + WorkspaceCommissioningFleetDependencyObserved { plan: _ } => WorkspaceExecutorPreparationRefused { reason: "commissioning dependency is unresolved" } WorkspaceCommissioningFleetRefused { reason: _ } => WorkspaceExecutorPreparationRefused { reason: "reviewed commissioning prestate is not established" } } } ProcessOutputAbsent => WorkspaceExecutorPreparationRefused { reason: "fleet generation does not admit the reviewed submission" } diff --git a/dag/gunbc/workspace_commissioning_fleet.dag b/dag/gunbc/workspace_commissioning_fleet.dag index 49a413c006d..7c513d5985b 100644 --- a/dag/gunbc/workspace_commissioning_fleet.dag +++ b/dag/gunbc/workspace_commissioning_fleet.dag @@ -41,6 +41,11 @@ import gunbc.runner_microvm_cell_readiness_store { read_cell_readiness_store } import gunbc.workspace_commissioning_apply_readback { workspace_commissioning_initial_ready, workspace_commissioning_credential_absent } import gunbc.workspace_commissioning_start_identity { workspace_commissioning_submitter_path } import gunbc.runner_microvm_slot_unit { workspace_launch_credential_source_path } +import product.host_identity { HostIdentity } +import gunbc.runner_microvm_network_sanitation_observe { slot_network_tun_queue_requirement } +import gunbc.host_kernel_dependency { HostKernelDependencyPlan, HostKernelCapabilitySatisfied, HostKernelDependencyPlanned, HostKernelDependencyRefused } +import gunbc.host_kernel_dependency_observe { host_tun_kernel_dependency_observe } +import gunbc.host_kernel_dependency_wire { host_kernel_dependency_row_prefix, host_kernel_dependency_from_text } // This is a read-only fleet subject, never a readiness permit or supply offer. The constructor // is available only here, after actual host and protected-store observations, not fixture input. @@ -53,6 +58,7 @@ type WorkspaceCommissioningFleetBasis sole_constructor { type WorkspaceCommissioningFleetObservation = WorkspaceCommissioningFleetObserved { basis: WorkspaceCommissioningFleetBasis } + | WorkspaceCommissioningFleetDependencyObserved { plan: HostKernelDependencyPlan } | WorkspaceCommissioningFleetRefused { reason: String } fn workspace_commissioning_basis_rows(basis: WorkspaceCommissioningFleetBasis) -> String { @@ -74,6 +80,25 @@ fn workspace_commissioning_candidate(host: NonEmptyStr, revision: CommitSha) -> } fn workspace_commissioning_candidate_slot(slot: RunnerSlotIdentity, revision: CommitSha) -> WorkspaceCommissioningFleetObservation { + match host_tun_kernel_dependency_observe(host: slot.host as HostIdentity, revision: revision, + requirement: slot_network_tun_queue_requirement(slot: slot)) { + HostKernelCapabilitySatisfied => workspace_commissioning_capable_candidate_slot(slot: slot, revision: revision) + HostKernelDependencyRefused { reason } => WorkspaceCommissioningFleetRefused { reason: reason } + HostKernelDependencyPlanned { plan } => { + let tree = Filesystem.Read(path: approval_broker_tree_receipt_path_for(root: microvm_controller_root, + revision: RevisionBoundAtEmission { revision: revision })) + match observe_broker_tree_receipt(success: tree.success, body: tree.content, error: tree.error as NonEmptyStr) { + TreeReceiptObserved { revision: actual, installed_digest: _ } => if actual == revision { + WorkspaceCommissioningFleetDependencyObserved { plan: plan } + } else { WorkspaceCommissioningFleetRefused { reason: "installed tree revision differs from dependency plan revision" } } + TreeReceiptUnreadable { detail: _ } => WorkspaceCommissioningFleetRefused { reason: "dependency observer installed tree receipt is unreadable" } + TreeReceiptMalformed { detail: _ } => WorkspaceCommissioningFleetRefused { reason: "dependency observer installed tree receipt is malformed" } + } + } + } +} + +fn workspace_commissioning_capable_candidate_slot(slot: RunnerSlotIdentity, revision: CommitSha) -> WorkspaceCommissioningFleetObservation { let installed = approval_broker_release_dir(root: microvm_controller_root, revision: RevisionBoundAtEmission { revision: revision }) match bind_slot_host(slot: slot) { MicrovmHostBound { spec: _, host: _ } => match resolve_network_subject(slot: slot) { @@ -211,7 +236,19 @@ fn workspace_commissioning_basis_observe(provenance: WorkspaceCommissioningProve fn workspace_commissioning_basis_from_privileged(text: String, host: NonEmptyStr, revision: CommitSha) -> WorkspaceCommissioningFleetObservation admit_callers: [decl_ref(module_path: "gunbc.workspace_commissioning_dispatch_transport", decl_name: "workspace_dispatch_observe")] { - match workspace_executor_plan_credential(plan_body: text) { + if starts_with(s: text, prefix: host_kernel_dependency_row_prefix) { + let slots = filter(workspace_execution_slot_identities(), slot => slot.host == host) + if slots.count() != 1 { WorkspaceCommissioningFleetRefused { reason: "dependency response has no unique designated slot" } } + else { match slots.first() { + Absent => WorkspaceCommissioningFleetRefused { reason: "dependency response slot is absent" } + Present { value: slot } => match host_kernel_dependency_from_text( + text: substring(s: text, start: length(host_kernel_dependency_row_prefix), end: length(text)), + host: host as HostIdentity, revision: revision, requirement: slot_network_tun_queue_requirement(slot: slot)) { + Present { value: plan } => WorkspaceCommissioningFleetDependencyObserved { plan: plan } + Absent => WorkspaceCommissioningFleetRefused { reason: "privileged kernel dependency response changed subject or is noncanonical" } + } + } } + } else { match workspace_executor_plan_credential(plan_body: text) { Present { value: credential } => match workspace_commissioning_directive_from_text(text: credential) { WorkspaceCommissioningDirectiveReady { provenance } => match workspace_executor_plan_prestate(plan_body: text) { Present { value: prestate } => { @@ -228,5 +265,5 @@ fn workspace_commissioning_basis_from_privileged(text: String, host: NonEmptyStr WorkspaceCommissioningDirectiveRefused { reason: _ } => WorkspaceCommissioningFleetRefused { reason: "privileged commissioning provenance unreadable" } } Absent => WorkspaceCommissioningFleetRefused { reason: "privileged commissioning response lacks its unique credential" } - } + } } } diff --git a/dag/test/claim/host_kernel_dependency_witness_test.dag b/dag/test/claim/host_kernel_dependency_witness_test.dag new file mode 100644 index 00000000000..6d97224ddb5 --- /dev/null +++ b/dag/test/claim/host_kernel_dependency_witness_test.dag @@ -0,0 +1,145 @@ +module test.claim.host_kernel_dependency_witness_test + +import std.types { Bool, String, NonEmptyStr, CommitSha } +import std.content_hash { serialize_content_hash } +import std.upsert_decision { Noop, Apply, Refuse } +import extdeps.linux.kernel { LinuxKernelRelease } +import extdeps.linux.tun_fdinfo { TunQueuesAttached, TunQueuesRefused, TunCensusQueueCountUnsupported, TunCensusQueueCountUnreadable, TunCensusReaderNotRoot, TunCensusMultiQueue, tun_attached_queue_requirement } +import extdeps.shell.exec { ProcessObservation } +import gunbc.fleet_host_identity { operator_host_srv1 } +import gunbc.runner_slot_allocation { workspace_execution_slot_identities } +import gunbc.runner_microvm_network_sanitation_observe { slot_network_tun_queue_requirement } +import gunbc.host_kernel_dependency { + UbuntuTunKernelProvider, HostKernelDependencyPlan, InstallKernelAndActivate, ActivateInstalledKernel, + HostKernelDependencyPlanned, HostKernelDependencyRefused, + KernelPackageExact, KernelPackageNeedsInstall, KernelPackageReadRefused, + tun_kernel_provider_for_host, tun_kernel_capability_decision, kernel_package_reading, + kernel_package_readings_join, host_kernel_dependency_plan, host_kernel_maintenance_capability, +} +import gunbc.host_kernel_dependency_observe { kernel_provider_platform_admits, kernel_provider_candidate_admits } +import gunbc.host_kernel_dependency_wire { host_kernel_dependency_text, host_kernel_dependency_from_text } +import gunbc.fleet_converge_plan { + WorkspaceCommissioningDependencyConverge, fleet_converge_plan_artifact_of_request, + fleet_converge_apply_artifact_admits, ApplyAwaitingCapability, +} +import gunbc.fleet_converge_terminal { AwaitingCapability } +import gunbc.workspace_commissioning_executor_plan { workspace_executor_scope_wire } + +// SUPPLIED AT THE BOUNDARY: observations below are adversarial inputs to the production +// dependency fold, not evidence about this machine. Actual ioctl execution has its own wet +// witness, and the installed privileged commissioning entry is exercised on srv1. +fn kernel_fixture_provider() -> UbuntuTunKernelProvider { + UbuntuTunKernelProvider { package: "linux-generic-hwe-24.04", version: "7.0.0-38.38~24.04.4", + architecture: "arm64", kernel_release: "7.0.0-38-generic" as LinuxKernelRelease } +} + +fn kernel_fixture_plan() -> HostKernelDependencyPlan { + HostKernelDependencyPlan { host: operator_host_srv1, + source_revision: "0123456789abcdef0123456789abcdef01234567" as CommitSha, + requirement: tun_attached_queue_requirement(name: "gunbc-tap13"), + running_release: "6.8.0-138-generic" as LinuxKernelRelease, boot_id: "old-boot", + provider: kernel_fixture_provider(), remedy: InstallKernelAndActivate, + package_simulation: "Inst linux-generic-hwe-24.04 (7.0.0-38.38~24.04.4 Ubuntu:24.04/noble-updates [arm64])\n" } +} + +test fn a_supported_query_never_demands_a_kernel_even_when_the_tap_is_held() -> Bool { + all([0, 1, 8], count => match tun_kernel_capability_decision( + requirement: tun_attached_queue_requirement(name: "gunbc-tap13"), reading: TunQueuesAttached { count: count }) { + Noop => true other => false + }) +} + +test fn only_unsupported_demands_the_selected_sanitation_operation() -> Bool { + let requirement = tun_attached_queue_requirement(name: "gunbc-tap13") + (match tun_kernel_capability_decision(requirement: requirement, reading: TunQueuesRefused { cause: TunCensusQueueCountUnsupported }) { + Apply { plan } => plan == requirement other => false + }) && all([TunCensusQueueCountUnreadable { cause: "permission denied" }, TunCensusReaderNotRoot { euid: 1000 }, TunCensusMultiQueue], cause => + match tun_kernel_capability_decision(requirement: requirement, reading: TunQueuesRefused { cause: cause }) { + Refuse { reason: _ } => true other => false + }) +} + +test fn installed_packages_select_activation_and_absent_packages_select_installation() -> Bool { + let p = kernel_fixture_plan() + (match host_kernel_dependency_plan(host: p.host, revision: p.source_revision, requirement: p.requirement, + running_release: p.running_release, boot_id: p.boot_id, provider: p.provider, packages: KernelPackageExact, simulation: "") { + HostKernelDependencyPlanned { plan } => plan.remedy == ActivateInstalledKernel other => false + }) && (match host_kernel_dependency_plan(host: p.host, revision: p.source_revision, requirement: p.requirement, + running_release: p.running_release, boot_id: p.boot_id, provider: p.provider, packages: KernelPackageNeedsInstall, simulation: p.package_simulation) { + HostKernelDependencyPlanned { plan } => plan.remedy == InstallKernelAndActivate other => false + }) && (match host_kernel_dependency_plan(host: p.host, revision: p.source_revision, requirement: p.requirement, + running_release: p.provider.kernel_release, boot_id: p.boot_id, provider: p.provider, packages: KernelPackageExact, simulation: "") { + HostKernelDependencyRefused { reason: _ } => true other => false + }) +} + +fn kernel_fixture_package_record() -> String { + "Package: linux-generic-hwe-24.04\nStatus: install ok installed\nArchitecture: arm64\nVersion: 7.0.0-38.38~24.04.4\n" +} + +test fn package_identity_and_read_failures_cannot_become_absence() -> Bool { + let p = kernel_fixture_provider() + (match kernel_package_reading(package: p.package, provider: p, + reading: ProcessObservation { exit_code: 0, stdout: kernel_fixture_package_record(), stderr: "" }) { + KernelPackageExact => true other => false + }) && (match kernel_package_reading(package: p.package, provider: p, + reading: ProcessObservation { exit_code: 1, stdout: "", stderr: "dpkg-query: no information is available about package linux-generic-hwe-24.04" }) { + KernelPackageNeedsInstall => true other => false + }) && all([ + "", "Status: hold ok not-installed\n", + "Package: linux-generic-hwe-24.04\nStatus: install ok installed\nArchitecture: amd64\nVersion: 7.0.0-38.38~24.04.4\n", + "Package: linux-generic-hwe-24.04\nStatus: install ok installed\nArchitecture: arm64\nVersion: 7.1.0\n", + concat(kernel_fixture_package_record(), "Version: 7.0.0-38.38~24.04.4\n"), + ], record => match kernel_package_reading(package: p.package, provider: p, + reading: ProcessObservation { exit_code: 0, stdout: record, stderr: "" }) { + KernelPackageReadRefused { reason: _ } => true other => false + }) && (match kernel_package_reading(package: p.package, provider: p, + reading: ProcessObservation { exit_code: 1, stdout: "", stderr: "permission denied" }) { + KernelPackageReadRefused { reason: _ } => true other => false + }) && (match kernel_package_readings_join(provider: p, readings: [KernelPackageExact, KernelPackageExact]) { + KernelPackageReadRefused { reason: _ } => true other => false + }) && (match kernel_package_readings_join(provider: p, readings: [KernelPackageNeedsInstall, KernelPackageReadRefused { reason: "denied" }, KernelPackageExact]) { + KernelPackageReadRefused { reason: _ } => true other => false + }) +} + +test fn kernel_provider_is_bound_to_platform_and_exact_package_metadata() -> Bool { + let p = kernel_fixture_provider() + tun_kernel_provider_for_host(host: operator_host_srv1) == Present { value: p } + && kernel_provider_platform_admits(os_release: "ID=ubuntu\nVERSION_ID=\"24.04\"\n", machine: "aarch64", provider: p) + && !kernel_provider_platform_admits(os_release: "ID=ubuntu\nVERSION_ID=24.04\n", machine: "x86_64", provider: p) + && !kernel_provider_platform_admits(os_release: "ID=debian\nVERSION_ID=24.04\n", machine: "aarch64", provider: p) + && !kernel_provider_platform_admits(os_release: "ID=ubuntu\nID=ubuntu\nVERSION_ID=24.04\n", machine: "aarch64", provider: p) + && kernel_provider_candidate_admits(provider: p, record: kernel_fixture_package_record()) + && !kernel_provider_candidate_admits(provider: p, record: concat(kernel_fixture_package_record(), "Architecture: arm64\n")) +} + +test fn dependency_response_is_bound_to_host_source_and_actual_operation() -> Bool { + let p = kernel_fixture_plan() + let wire = host_kernel_dependency_text(plan: p) + host_kernel_dependency_from_text(text: wire, host: p.host, revision: p.source_revision, requirement: p.requirement) == Present { value: p } + && host_kernel_dependency_from_text(text: concat(wire, "\n"), host: p.host, revision: p.source_revision, requirement: p.requirement) == none + && host_kernel_dependency_from_text(text: wire, host: p.host, revision: "1123456789abcdef0123456789abcdef01234567" as CommitSha, requirement: p.requirement) == none + && host_kernel_dependency_from_text(text: wire, host: p.host, revision: p.source_revision, requirement: tun_attached_queue_requirement(name: "gunbc-tap12")) == none +} + +// Production fleet artifact and admission, not a second renderer. Pending host work cannot +// produce a commissioning credential, an admitted apply, or a successful apply terminal. +test fn commissioning_plan_exposes_the_dependency_without_publishing_readiness() -> Bool { + let request = WorkspaceCommissioningDependencyConverge { plan: kernel_fixture_plan() } + let artifact = fleet_converge_plan_artifact_of_request(request: request, prior_generation: 20) + (match artifact.apply_terminal { AwaitingCapability { capability } => capability == host_kernel_maintenance_capability other => false }) + && string_contains(s: artifact.plan_body, pattern: "install-package:linux-generic-hwe-24.04=7.0.0-38.38~24.04.4") + && string_contains(s: artifact.plan_body, pattern: "activate-kernel:7.0.0-38-generic") + && !string_contains(s: artifact.plan_body, pattern: "commissioning-credential:") + && !string_contains(s: artifact.apply_shell, pattern: "apt-get") + && !string_contains(s: artifact.apply_shell, pattern: "systemctl reboot") + && (match fleet_converge_apply_artifact_admits(planned_host_text: "srv1", planned_scope_wire: workspace_executor_scope_wire, + planned_fingerprint_text: serialize_content_hash(hash: artifact.subject.member_set_fingerprint), observed: request) { + ApplyAwaitingCapability { capability } => capability == host_kernel_maintenance_capability other => false + }) + && (match filter(workspace_execution_slot_identities(), slot => slot.host == "srv1").first() { + Present { value: slot } => slot_network_tun_queue_requirement(slot: slot) == kernel_fixture_plan().requirement + Absent => false + }) +} diff --git a/dag/test/claim/workspace_network_observation_witness_test.dag b/dag/test/claim/workspace_network_observation_witness_test.dag index eafd498acfd..6c1b07e51d8 100644 --- a/dag/test/claim/workspace_network_observation_witness_test.dag +++ b/dag/test/claim/workspace_network_observation_witness_test.dag @@ -3,7 +3,7 @@ module test.claim.workspace_network_observation_witness_test import std.types { Bool, String } import std.optional { Present, Absent } import extdeps.linux.procfs { procfs_status_effective_uid } -import extdeps.ethtool { ethtool_combined_channels, EthtoolCombinedChannelsCurrent, EthtoolCombinedChannelsUnreadable } +import extdeps.linux.ethtool_ioctl { linux_ethtool_channels_decode, LinuxEthtoolChannelsRead, LinuxEthtoolChannelsUnsupported, LinuxEthtoolChannelsUnreadable } import extdeps.linux.tun_fdinfo { TunDescriptorCount, TunDescriptorUnreadable, TunCensusMultiQueue, TunCensusFlagsUnreadable, TunCensusProcUnreadable, @@ -18,12 +18,6 @@ import extdeps.linux.tun_fdinfo { // floor. The real reader runs in // test.claim.workspace_network_real_reader_witness_test on the local-repo wet lane. -data channels_one: String = "Channel parameters for tap0:\nPre-set maximums:\nRX:\t\tn/a\nTX:\t\tn/a\nOther:\t\tn/a\nCombined:\t1\nCurrent hardware settings:\nRX:\t\tn/a\nTX:\t\tn/a\nOther:\t\tn/a\nCombined:\t1\n" - -data channels_current_na: String = "Channel parameters for tap0:\nPre-set maximums:\nRX:\t\tn/a\nTX:\t\tn/a\nOther:\t\tn/a\nCombined:\t1\nCurrent hardware settings:\nRX:\t\tn/a\nTX:\t\tn/a\nOther:\t\tn/a\nCombined:\tn/a\n" - -data channels_maximums_only: String = "Channel parameters for tap0:\nPre-set maximums:\nRX:\t\tn/a\nTX:\t\tn/a\nOther:\t\tn/a\nCombined:\t1\n" - // Positive control: no holder descriptor, no attached queue, two clean sweeps. test fn zero_holders_requires_every_observation_to_affirm_zero() -> Bool { match tun_census_join(before: TunQueuesAttached { count: 0 }, first: TunSweepComplete { holders: [], exited: 2 }, @@ -74,17 +68,22 @@ test fn multi_queue_and_malformed_flags_refuse() -> Bool { && (match tun_queue_mode(flags: "0x1102\n") { TunMultiQueue => true other => false }) && (match tun_queue_mode(flags: "1002") { TunFlagsMalformed => true other => false }) && (match tun_queue_mode(flags: "0xzz") { TunFlagsMalformed => true other => false }) - && (match tun_queue_reading(mode: TunMultiQueue, channels: EthtoolCombinedChannelsCurrent { count: 0 }) { TunQueuesRefused { cause: TunCensusMultiQueue } => true other => false }) - && (match tun_queue_reading(mode: TunFlagsMalformed, channels: EthtoolCombinedChannelsCurrent { count: 0 }) { TunQueuesRefused { cause: TunCensusFlagsUnreadable } => true other => false }) + && (match tun_queue_reading(mode: TunMultiQueue, channels: LinuxEthtoolChannelsRead { combined: 0 }) { TunQueuesRefused { cause: TunCensusMultiQueue } => true other => false }) + && (match tun_queue_reading(mode: TunFlagsMalformed, channels: LinuxEthtoolChannelsRead { combined: 0 }) { TunQueuesRefused { cause: TunCensusFlagsUnreadable } => true other => false }) } test fn channel_count_reads_only_the_current_combined_count() -> Bool { - (match ethtool_combined_channels(device: "tap0", success: true, value: channels_one) { EthtoolCombinedChannelsCurrent { count: n } => n == 1 other => false }) - && (match ethtool_combined_channels(device: "tap0", success: true, value: channels_current_na) { EthtoolCombinedChannelsUnreadable { cause: _ } => true other => false }) - && (match ethtool_combined_channels(device: "tap0", success: true, value: channels_maximums_only) { EthtoolCombinedChannelsUnreadable { cause: _ } => true other => false }) - && (match ethtool_combined_channels(device: "tap0", success: false, value: channels_one) { EthtoolCombinedChannelsUnreadable { cause: _ } => true other => false }) - && (match ethtool_combined_channels(device: "tap1", success: true, value: channels_one) { EthtoolCombinedChannelsUnreadable { cause: _ } => true other => false }) - && (match ethtool_combined_channels(device: "tap0", success: true, value: "") { EthtoolCombinedChannelsUnreadable { cause: _ } => true other => false }) + (match linux_ethtool_channels_decode(device: "tap0", text: "channels:tap0:60:8:8:8:16:2:2:2:1") { LinuxEthtoolChannelsRead { combined: n } => n == 1 other => false }) + && (match linux_ethtool_channels_decode(device: "tap0", text: "channels:tap0:60:8:8:8:16:2:2:2") { LinuxEthtoolChannelsUnreadable { cause: _ } => true other => false }) + && (match linux_ethtool_channels_decode(device: "tap0", text: "channels:tap0:60:8:8:8:16:2:2:2:-1") { LinuxEthtoolChannelsUnreadable { cause: _ } => true other => false }) + && (match linux_ethtool_channels_decode(device: "tap0", text: "channels:tap1:60:8:8:8:16:2:2:2:0") { LinuxEthtoolChannelsUnreadable { cause: _ } => true other => false }) + && (match linux_ethtool_channels_decode(device: "tap0", text: "channels:tap0:61:8:8:8:16:2:2:2:0") { LinuxEthtoolChannelsUnreadable { cause: _ } => true other => false }) +} + +test fn only_the_kernel_unsupported_errno_establishes_absent_capability() -> Bool { + (match linux_ethtool_channels_decode(device: "tap0", text: "errno:tap0:95") { LinuxEthtoolChannelsUnsupported => true other => false }) + && all(["errno:tap0:1", "errno:tap0:13", "errno:tap0:19", "errno:tap1:95", "netlink error: Operation not supported", "", "errno:tap0:95\nerrno:tap0:1"], text => + match linux_ethtool_channels_decode(device: "tap0", text: text) { LinuxEthtoolChannelsUnreadable { cause: _ } => true other => false }) } test fn fdinfo_names_exactly_this_interface() -> Bool { diff --git a/dag/test/claim/workspace_network_real_reader_witness_test.dag b/dag/test/claim/workspace_network_real_reader_witness_test.dag index fde277bb90a..8e4d2f78b57 100644 --- a/dag/test/claim/workspace_network_real_reader_witness_test.dag +++ b/dag/test/claim/workspace_network_real_reader_witness_test.dag @@ -5,6 +5,7 @@ import std.optional { Present, Absent } import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } import extdeps.filesystem.filesystem_io { Filesystem } import extdeps.linux.procfs { procfs_status_effective_uid } +import extdeps.linux.ethtool_ioctl { linux_ethtool_channels_observe, LinuxEthtoolChannelsUnsupported, LinuxEthtoolChannelsUnreadable } import extdeps.linux.tun_fdinfo { TunInterfaceMissing, TunDescriptorUnreadable, TunCensusReaderNotRoot, TunCensusSubjectInadmissible, tun_descriptor_census_observe @@ -22,6 +23,8 @@ data live_tree_disposition: LiveTreeDisposition = SubstrateInputsOnly // effective uid dictates, and this claim reads that uid independently. As root, the reader reaches // tun_flags and answers Missing. Otherwise it refuses on the reader identity before touching /proc. // Neither route may produce a count. +// Execute the actual Python/native-ioctl boundary too. Loopback exists but has no channel +// operation; the absent interface answers ENODEV, which must NOT demand a kernel upgrade. test fn real_reader_takes_the_route_the_reader_identity_dictates() -> Bool { let status = Filesystem.Read(path: "/proc/self/status") status.success && (match procfs_status_effective_uid(content: status.content) { @@ -32,4 +35,6 @@ test fn real_reader_takes_the_route_the_reader_identity_dictates() -> Bool { other => false } }) && (match tun_descriptor_census_observe(name: "bad/name") { TunDescriptorUnreadable { cause: TunCensusSubjectInadmissible } => true other => false }) + && (match linux_ethtool_channels_observe(device: "lo") { LinuxEthtoolChannelsUnsupported => true other => false }) + && (match linux_ethtool_channels_observe(device: "gunbcnotap0") { LinuxEthtoolChannelsUnreadable { cause: _ } => true other => false }) } diff --git a/dag/test/claim/workspace_workflow_dispatch_witness_test.dag b/dag/test/claim/workspace_workflow_dispatch_witness_test.dag index 24bf2a28401..db2ebeffc1c 100644 --- a/dag/test/claim/workspace_workflow_dispatch_witness_test.dag +++ b/dag/test/claim/workspace_workflow_dispatch_witness_test.dag @@ -3,7 +3,7 @@ module test.claim.workspace_workflow_dispatch_witness_test import std.types { Bool, String } import gunbc.fleet_converge_workflow { WorkspaceAllocationPlan, WorkspaceImageBuild, - fleet_converge_job, fleet_converge_shared_job_if, fleet_converge_dispatch_inputs, fleet_converge_mode_scope, fleet_converge_mode_job_id, fleet_converge_shared_job_id, + fleet_converge_job_with_steps, fleet_converge_shared_job_if, fleet_converge_dispatch_inputs, fleet_converge_mode_scope, fleet_converge_mode_job_id, fleet_converge_shared_job_id, fleet_converge_mode_release_product, CompilerPairOnly, fleet_converge_mode_fleet_ssh_key_demand, FleetSshKeyNotConsumed, } @@ -41,10 +41,11 @@ test fn fleet_dispatch_stays_within_github_input_limit() -> Bool { count(fleet_converge_dispatch_inputs) <= 25 } -// Exercise the actual emitted job. Shared modes must select it positively; dedicated modes -// cannot consume another runner or attempt the shared cloud login as a side effect. +// SUPPLIED AT THE BOUNDARY: an empty step list avoids constructing unrelated deployment +// programs. The production job envelope still owns the actual if-condition assignment. Shared +// modes must select it positively; dedicated modes cannot select the credentialed shared job. test fn dedicated_installs_do_not_also_run_the_shared_credentialed_job() -> Bool { - match fleet_converge_job().if_condition { + match fleet_converge_job_with_steps(steps: []).if_condition { Absent => false Present { value: condition } => condition == fleet_converge_shared_job_if && string_contains(s: condition, pattern: "github.event.inputs.mode == 'workspace_image_build'") diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index f7ce1ccb2b4..245b0e748cc 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -5,10 +5,12 @@ allocation, controller and cleanup protocols. No customer create API, additional hosts, CI-runner migration, or automatic expiry scheduling is part of this change. Current status: the pinned image/controller, network receipt and slot resource -boundary have passed live readback. Commissioning refuses because srv1's running -kernel cannot report attached TUN queues. A concrete kernel maintenance proposal -is recorded below and has not been applied. No VM boot or authenticated guest SSH -is yet proven. Guest access uses the dedicated key already on srv1. +boundary have passed live readback. srv1's running kernel cannot report attached +TUN queues. Commissioning now models that capability dependency and derives a +kernel installation/activation proposal in its existing convergence scope. Host +maintenance actuation remains an explicit `AwaitingCapability` frontier; no kernel +installation or reboot has been applied. No VM boot or authenticated guest SSH is +yet proven. Guest access uses the dedicated key already on srv1. ## Observed gaps @@ -419,6 +421,39 @@ has been commissioned or booted. ### Proposed host maintenance (not applied) +The commissioning planner derives this proposal from the selected sanitation +reader's `TunAttachedQueueRequirement`. Its observation now invokes Linux's +`SIOCETHTOOL / ETHTOOL_GCHANNELS` ABI directly through the declared Python runtime +boundary. Only numeric `EOPNOTSUPP` demands a provider. Permission failures, +missing devices, malformed replies, a failed reader or multi-queue TAPs refuse; +they cannot become a reason to upgrade. A successful count on a backported kernel +needs no package lookup. A successful nonzero count establishes capability but +still fails the unchanged sanitation census. + +The host policy in `gunbc.host_kernel_dependency` selects the Ubuntu provider; +VM code does not select a version. Readback binds the proposal to host, source +revision, interface, running release and boot identity. It checks the distribution, +architecture, installed package identities, exact available candidate and a +non-removing apt simulation. If the selected image/modules are already installed +and their boot artifacts are readable, the proposal contains activation only. +If the selected kernel already runs but the operation remains unsupported, it +refuses instead of entering a reboot loop. + +The existing `workspace-slot-commissioning` plan carries the dependency, package +simulation and continuation. It cannot mint a commissioning credential or initial +readiness. Its apply admission is `AwaitingCapability`: plan-bound interruption +authority, runner drain, verified console recovery and an external boot observer +are not established for srv1. `gunbc.host_reset_boot_selection` currently admits +only the separately enrolled reset subject, not srv1. This change implements +capability-driven dependency planning; it does **not** implement an unattended +srv1 kernel installer/reboot actuator. After an admitted maintenance path establishes +the running capability, rerunning the same commissioning goal continues through +the full sanitation, protected-state and runtime checks. + +This is ordinary domain composition using the existing `ensure` decision and +fleet plan, not a claim that the generic completion/elaboration engine proposed +in `docs/plans/ensure-closure-design.md` has been implemented. + The read-only package simulation on srv1 selects Ubuntu's arm64 HWE meta-package `linux-generic-hwe-24.04=7.0.0-38.38~24.04.4`. It adds eight packages, upgrades none and removes none: the image, modules, ZFS module package, header packages @@ -465,3 +500,37 @@ This validates refusal reporting against the installed release; it is not a new controller installation, commissioning apply or ongoing CI claim. Independent post-readback checks find no allocation, commissioning or readiness files, and the historical controller remains stopped with `MainPID=0` and no queued job. + +### Dependency observation evidence + +The production `workspace_dispatch_plan_cli`, run read-only against release +`4fa60f71342` from source snapshot SHA-256 +`9aeb7a306baf7e08d7a6d81801a7b4ed02f3e04ef6869c105ef8754b30593b45`, +returns exit 0 and a canonical `commissioning-kernel-dependency` response for +`srv1 / gunbc-tap13`. The response binds running release `6.8.0-138-generic`, +the observed boot ID and selected `7.0.0-38-generic` provider. Its live apt +simulation reports eight new packages, zero upgrades and zero removals. This is +a successful **plan observation**, not completed commissioning or an installation +of the new source. The log and decoded response are +`target/srv1-bringup-evidence/kernel-dependency-live-r2.{log,json}`. + +The Python ABI program rendered by `linux_ethtool_channels_program` was also run +read-only on srv1. `gunbc-tap13` returned numeric errno 95; `gunbcnotap0` returned +19; the physical `enP3p3s0f1` returned a combined count of 63, independently +matching `ethtool --show-channels`. The receipt is +`target/srv1-bringup-evidence/ethtool-ioctl-live.txt`. The declared wet witness +executes the same native reader and distinguishes unsupported from missing. + +The initial success-reply decoder witness exposed the seed's already-recorded +`optional_equality_answers_by_representation` defect. The consumer now eliminates +the optional by a match, following that ledger's existing discipline; it does not +claim to repair the interpreter. The dependency, wire binding, fleet admission, +network producer and dispatch controls pass. The workflow test responsible for +CI run `37915381374`'s failure previously constructed every unrelated step merely +to inspect the job condition. The production envelope now accepts its step list; +the witness supplies an empty list at that boundary and still checks the actual +condition assignment. It passes at 3,142 evaluation steps, versus the failed +run's 446,125, below the new-witness budget. Workflow behavior is unchanged. + +Post-observation readback still finds `6.8.0-138-generic` running, no allocation, +commissioning or readiness files, and controller `MainPID=0` with no queued job. From 58019fe745f5135fba98a3c2be1ee3029d3bf0d5 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Fri, 9 Oct 2026 17:36:40 +0000 Subject: [PATCH 19/19] Realize reviewed kernel dependencies through guarded host maintenance --- dag/extdeps/bootloader/grub.dag | 13 ++ dag/extdeps/package_managers/apt.dag | 11 +- dag/extdeps/systemd/systemctl.dag | 17 ++ dag/gunbc/host/host_kernel_boot_selection.dag | 24 ++ dag/gunbc/host/host_kernel_dependency.dag | 3 +- .../host/host_kernel_dependency_apply.dag | 209 ++++++++++++++++++ ...tics_instead_of_demanding_its_provider.dag | 1 + ...ost_kernel_boot_selection_witness_test.dag | 16 ++ .../host_kernel_maintenance_witness_test.dag | 81 +++++++ docs/plans/srv1-workspace-vm-bringup.md | 81 +++++-- 10 files changed, 438 insertions(+), 18 deletions(-) create mode 100644 dag/gunbc/host/host_kernel_boot_selection.dag create mode 100644 dag/gunbc/host/host_kernel_dependency_apply.dag create mode 100644 dag/test/claim/host_kernel_boot_selection_witness_test.dag create mode 100644 dag/test/claim/host_kernel_maintenance_witness_test.dag diff --git a/dag/extdeps/bootloader/grub.dag b/dag/extdeps/bootloader/grub.dag index e1e351e4ae8..7508600afb4 100644 --- a/dag/extdeps/bootloader/grub.dag +++ b/dag/extdeps/bootloader/grub.dag @@ -81,6 +81,19 @@ fn grub_config_render(cfg: GrubConfig) -> String { join([header, body], "\n") } +// The first linux/initrd command's literal operand in a generated configuration. +// This is a narrow read, not an interpreter for arbitrary GRUB scripts: computed +// or quoted operands remain different from the literal path a caller expects. +fn grub_first_command_operand(config: String, command: String) -> String? { + let rows = map(split(s: config, delimiter: "\n"), line => + filter(flat_map(split(s: trim(s: line), delimiter: "\t"), part => split(s: part, delimiter: " ")), word => word != "")) + let commands = filter(rows, words => match words.first() { Present { value: word } => word == command Absent => false }) + match commands.first() { + Absent => none + Present { value: words } => words.skip(n: 1).first() + } +} + data grub_console_banner_prefix: NonEmptyStr = "GNU GRUB version " data grub_failed_default_and_fallback: NonEmptyStr = "Failed to boot both default and fallback entries." data grub_error_prefix: NonEmptyStr = "error: " diff --git a/dag/extdeps/package_managers/apt.dag b/dag/extdeps/package_managers/apt.dag index a5e7eb5ae1e..c72b6d75fad 100644 --- a/dag/extdeps/package_managers/apt.dag +++ b/dag/extdeps/package_managers/apt.dag @@ -62,7 +62,8 @@ data apt_man_url: NonEmptyStr = "https://manpages.debian.org/testing/apt/apt.8.e // apt_install_argv was DELETED here for the reason dpkg.dag's Status row states: it was this // Install row's argv authored a second time, and its last consumer now binds `package` against -// this declaration instead. +// this declaration instead. InstallExactNonRemoving follows SimulateInstall's selection and +// removal policy and keeps needrestart in list-only mode during supervised host maintenance. service apt.PackageManager { operation ShowExact { input { selection: NonEmptyStr } @@ -95,4 +96,12 @@ service apt.PackageManager { nonzero => String "apt-get install failed" } } + operation InstallExactNonRemoving { + requires Network + input { selection: NonEmptyStr } + output { success: Bool from "exit_success", stdout: String from "stdout", stderr: String from "stderr" } + transport shell { argv: ["env", "LC_ALL=C", "DEBIAN_FRONTEND=noninteractive", "NEEDRESTART_MODE=l", "apt-get", "--yes", "--no-install-recommends", "--no-remove", "install", "{selection}"] } + exit { 0 => Unit nonzero => String "apt-get exact non-removing install refused" } + mock_response { 1 => { success: false, stdout: "", stderr: "no installation supplied" } "no fabricated package installation" } + } } diff --git a/dag/extdeps/systemd/systemctl.dag b/dag/extdeps/systemd/systemctl.dag index 1acf1739c59..8724692fe47 100644 --- a/dag/extdeps/systemd/systemctl.dag +++ b/dag/extdeps/systemd/systemctl.dag @@ -442,6 +442,8 @@ type UnitFileState // actually executes. The trigger that closes it again is argv element evaluation in the // materializer, which `std.operation_argv` already carries as declared residue. // +// MaskRuntime follows systemctl(1), v255: --runtime places a mask under /run and +// without --now the active incarnation continues. It does not stop a workload. service systemd.Systemctl { operation Enable { requires Network @@ -573,6 +575,21 @@ service systemd.Systemctl { } } + operation MaskRuntime { + input { unit: NonEmptyStr } + output { success: Bool from "exit_success", stdout: String from "stdout", stderr: String from "stderr" } + transport shell { argv: ["systemctl", "mask", "--runtime", "{unit}"] } + exit { 0 => Unit nonzero => String "runtime mask refused" } + mock_response { 1 => { success: false, stdout: "", stderr: "no mask supplied" } "no fabricated inhibition" } + } + + operation Reboot { + output { success: Bool from "exit_success", stderr: String from "stderr" } + transport shell { argv: ["systemctl", "reboot"] } + exit { 0 => Unit nonzero => String "normal reboot request refused" } + mock_response { 1 => { success: false, stderr: "no reboot supplied" } "no fabricated activation" } + } + operation SetProperty { requires Network input { unit: NonEmptyStr, property: NonEmptyStr, value: NonEmptyStr } diff --git a/dag/gunbc/host/host_kernel_boot_selection.dag b/dag/gunbc/host/host_kernel_boot_selection.dag new file mode 100644 index 00000000000..7cb28fceb76 --- /dev/null +++ b/dag/gunbc/host/host_kernel_boot_selection.dag @@ -0,0 +1,24 @@ +module gunbc.host_kernel_boot_selection + +import std.types { Bool, String } +import std.algebra { trim } +import extdeps.bootloader.grub { grub_first_command_operand } + +// srv1's observed Ubuntu GRUB realization uses literal default 0 and the next_entry +// override. Refuse a different realization or a pending override instead of guessing its +// boot selection. This changes no boot policy and retains the old kernels for BMC recovery. +fn kernel_maintenance_grub_selects(config: String, environment: String, target: String) -> Bool { + let defaults = filter(map(split(s: config, delimiter: "\n"), line => trim(s: line)), line => starts_with(s: line, prefix: "set default=")) + let overrides = ["next_entry", "prev_entry", "prev_saved_entry", "initrdfail"] + let next_entry_reference = "$\{next_entry}" + defaults.count() == 2 && any(defaults, line => line == "set default=\"0\"") + && any(defaults, line => line == concat("set default=\"", concat(next_entry_reference, "\""))) + && !any(split(s: environment, delimiter: "\n"), line => any(overrides, key => + starts_with(s: line, prefix: concat(key, "=")) && line != concat(key, "="))) + && (match grub_first_command_operand(config: config, command: "linux") { + Present { value: path } => path == concat("/vmlinuz-", target) Absent => false + }) + && (match grub_first_command_operand(config: config, command: "initrd") { + Present { value: path } => path == concat("/initrd.img-", target) Absent => false + }) +} diff --git a/dag/gunbc/host/host_kernel_dependency.dag b/dag/gunbc/host/host_kernel_dependency.dag index f33539b7579..49d9576bc99 100644 --- a/dag/gunbc/host/host_kernel_dependency.dag +++ b/dag/gunbc/host/host_kernel_dependency.dag @@ -147,7 +147,8 @@ fn host_kernel_dependency_action_lines(plan: HostKernelDependencyPlan) -> List CliWireResponse { + CliWirePrintable { bytes: "", exit: exit_failure(reason: reason) } +} + +fn kernel_maintenance_report(text: String) -> CliWireResponse { + CliWirePrintable { bytes: concat(text, "\n"), exit: ExitSuccess } +} + +fn kernel_maintenance_plan_read(text: String) -> HostKernelDependencyPlan? { + match parse_json_document(s: text) { + JsonDocumentUnreadable { gap: _ } => none + JsonDocumentParsed { value: doc } => match kernel_dependency_field(doc: doc, name: "host") { + Absent => none + Present { value: host } => match kernel_dependency_field(doc: doc, name: "source_revision") { + Absent => none + Present { value: revision } => { + let slots = filter(workspace_execution_slot_identities(), slot => (slot.host as String) == host) + if !content_hash_validate_lower_hex_length(text: revision, expected_hex_digits: 40) || slots.count() != 1 { none } + else { match slots.first() { + Absent => none + Present { value: slot } => host_kernel_dependency_from_text(text: text, host: slot.host, + revision: revision as CommitSha, requirement: slot_network_tun_queue_requirement(slot: slot)) + } } + } + } + } + } +} + +// An install may advance only package state, not the host, boot, source, requirement or +// provider. Activation never uses an installation simulation as evidence of installed bytes. +fn kernel_maintenance_plan_continues(reviewed: HostKernelDependencyPlan, current: HostKernelDependencyPlan, + action: HostKernelMaintenanceAction) -> Bool { + let same_subject = reviewed.host == current.host && reviewed.source_revision == current.source_revision + && reviewed.requirement == current.requirement && reviewed.running_release == current.running_release + && reviewed.boot_id == current.boot_id && reviewed.provider == current.provider + same_subject && (host_kernel_dependency_text(plan: reviewed) == host_kernel_dependency_text(plan: current) + || ((action == KernelInstall || action == KernelActivate) && current.remedy == ActivateInstalledKernel)) +} + +// list-units is a census, so malformed/foreign rows refuse the entire reading. All loaded +// runner instances are included, including historical masked slots; no desired-width guess +// may omit a still-running service. The mask leaves each ephemeral runner's current job alone. +fn kernel_maintenance_runner_units(host: HostIdentity, text: String) -> List? { + let lines = filter(split(s: text, delimiter: "\n"), line => trim(s: line) != "") + let names = map(lines, line => match filter(split(s: trim(s: line), delimiter: " "), word => word != "").first() { + Present { value: name } => name + Absent => "" + }) + if names.count() == 0 || !all(names, name => census_unit_name_is_slot(name: name) + && unit_is_owned_by_host(host: host, unit: name) + && count(filter(names, other => other == name)) == 1) { none } + else { Present { value: map(names, name => name as NonEmptyStr) } } +} + +// Explicit lifecycle vocabulary: an unknown state is not quiescence. The kernel's existing +// cgroup.events observer covers descendants; a service state alone cannot establish drain. +fn kernel_maintenance_unit_quiet(unit: NonEmptyStr, require_mask: Bool) -> Bool { + let state = systemd.Systemctl.ShowProperty(unit: unit, property: "ActiveState") + let mask = systemd.Systemctl.ShowProperty(unit: unit, property: "UnitFileState") + let job = systemd.Systemctl.ShowProperty(unit: unit, property: "Job") + state.success && mask.success && job.success && kernel_maintenance_unit_readback_admits( + state: trim(s: state.value), mask: trim(s: mask.value), job: trim(s: job.value), require_mask: require_mask, + occupancy: observe_slot_cgroup_occupancy(transport: LocalShell, unit: unit)) +} + +fn kernel_maintenance_unit_readback_admits(state: String, mask: String, job: String, + require_mask: Bool, occupancy: CgroupOccupancy) -> Bool { + job == "" && (state == "inactive" || state == "failed") + && (!require_mask || mask == "masked" || mask == "masked-runtime") + && (match occupancy { + CgroupAbsent => true CgroupObservedEmpty => true + CgroupObservedPopulated => false CgroupUnobserved { cause: _ } => false + }) +} + +fn kernel_maintenance_drain(plan: HostKernelDependencyPlan, units: List) -> CliWireResponse { + let results = map(units, unit => { + let before = systemd.Systemctl.ShowProperty(unit: unit, property: "UnitFileState") + if !before.success { false } + else if trim(s: before.value) == "masked" || trim(s: before.value) == "masked-runtime" { true } + else { systemd.Systemctl.MaskRuntime(unit: unit).success } + }) + if !all(results, ok => ok) { kernel_maintenance_refused(reason: "runner inhibition incomplete; read back the runtime masks before retrying; no package or reboot effect ran") } + else if all(units, unit => kernel_maintenance_unit_quiet(unit: unit, require_mask: true)) { + kernel_maintenance_report(text: "kernel-maintenance: runners drained; runtime masks expire at reboot") + } else { kernel_maintenance_refused(reason: "kernel-maintenance: awaiting current runner jobs and their cgroups; replacement inhibited, no jobs stopped") } +} + +fn kernel_maintenance_activate(plan: HostKernelDependencyPlan) -> CliWireResponse { + let config = Filesystem.Read(path: "/boot/grub/grub.cfg") + let environment = Filesystem.Read(path: "/boot/grub/grubenv") + let boot = Filesystem.Read(path: "/proc/sys/kernel/random/boot_id") + if !config.success || !environment.success || !boot.success || trim(s: boot.content) != (plan.boot_id as String) + || !kernel_maintenance_grub_selects(config: config.content, environment: environment.content, target: plan.provider.kernel_release as String) { + kernel_maintenance_refused(reason: "reviewed boot identity or GRUB selection of the exact image/initramfs is not established") + } else { + let reboot = systemd.Systemctl.Reboot() + if reboot.success { kernel_maintenance_report(text: "kernel-maintenance: normal reboot requested; peer must verify a new boot, target kernel and actual queue capability") } + else { kernel_maintenance_refused(reason: "reboot acknowledgement unavailable; peer must read boot identity before any retry") } + } +} + +fn kernel_maintenance_apply_observed(plan: HostKernelDependencyPlan, current: HostKernelDependencyPlan, + action: HostKernelMaintenanceAction) -> CliWireResponse { + let census = systemd.Systemctl.ListUnitsAllLoaded(pattern: "actions*runner*.service") + if !census.success { kernel_maintenance_refused(reason: "runner population unreadable") } + else { match kernel_maintenance_runner_units(host: plan.host, text: census.stdout) { + Absent => kernel_maintenance_refused(reason: "runner population is empty, malformed or includes a foreign unit") + Present { value: units } => if (action == KernelDrain) { kernel_maintenance_drain(plan: plan, units: units) } + else if !all(units, unit => kernel_maintenance_unit_quiet(unit: unit, require_mask: true)) { + kernel_maintenance_refused(reason: "kernel maintenance requires every runner inhibited and drained, including its cgroup") + } else { + let slots = filter(workspace_execution_slot_identities(), slot => slot.host == plan.host) + let quiet = all(slots, slot => kernel_maintenance_unit_quiet( + unit: microvm_slot_unit_instance_name(slot_instance: runner_slot_instance_name(slot: slot) as String), require_mask: false)) + if !quiet { kernel_maintenance_refused(reason: "workspace controller is active or its workload is unreadable") } + else if (action == KernelInstall) { match current.remedy { + ActivateInstalledKernel => kernel_maintenance_report(text: "kernel-maintenance: exact kernel and boot artifacts already installed; activation pending") + InstallKernelAndActivate => { + let installed = apt.PackageManager.InstallExactNonRemoving(selection: kernel_provider_selection(provider: plan.provider)) + if !installed.success { kernel_maintenance_refused(reason: concat("kernel package installation did not complete; re-observe before retry: ", installed.stderr)) } + else { match host_tun_kernel_dependency_observe(host: plan.host, revision: plan.source_revision, requirement: plan.requirement) { + HostKernelDependencyPlanned { plan: after } => if after.remedy == ActivateInstalledKernel + && kernel_maintenance_plan_continues(reviewed: plan, current: after, action: KernelInstall) { + kernel_maintenance_report(text: "kernel-maintenance: exact installed packages, boot image, initramfs and TUN implementation read back; activation pending") + } else { kernel_maintenance_refused(reason: "package command succeeded but installed kernel readback did not establish the reviewed target") } + HostKernelDependencyRefused { reason } => kernel_maintenance_refused(reason: reason) + HostKernelCapabilitySatisfied => kernel_maintenance_refused(reason: "running capability changed during installation; fresh observation required") + } } + } + } } else if action == KernelActivate && (current.remedy == ActivateInstalledKernel) { + kernel_maintenance_activate(plan: plan) + } else { kernel_maintenance_refused(reason: "activation requires installed kernel readback; verification requires a new capable boot") } + } + } } +} + +fn kernel_maintenance_cli(action: HostKernelMaintenanceAction) -> CliWireResponse { + let self = Filesystem.Read(path: "/proc/self/status") + let root = self.success && (match procfs_status_effective_uid(content: self.content) { Present { value: uid } => uid == 0 Absent => false }) + let input = Filesystem.Read(path: "/dev/stdin") + if !root || !input.success { kernel_maintenance_refused(reason: "kernel maintenance requires an explicit administrator invocation and reviewed plan on stdin") } + else { match kernel_maintenance_plan_read(text: trim(s: input.content)) { + Absent => kernel_maintenance_refused(reason: "kernel maintenance input is not the canonical dependency for an enrolled workspace host") + Present { value: plan } => match host_tun_kernel_dependency_observe(host: plan.host, + revision: plan.source_revision, requirement: plan.requirement) { + HostKernelDependencyRefused { reason } => kernel_maintenance_refused(reason: reason) + HostKernelCapabilitySatisfied => { + let boot = Filesystem.Read(path: "/proc/sys/kernel/random/boot_id") + let running = kernel_read_command(command: uname_release_command()) + let target = match running { Present { value: running_kernel } => running_kernel == (plan.provider.kernel_release as String) Absent => false } + if (action != KernelVerify) { kernel_maintenance_report(text: "kernel-maintenance: required capability already satisfied; no mutation") } + else if boot.success && trim(s: boot.content) != "" && trim(s: boot.content) != (plan.boot_id as String) && target { + kernel_maintenance_report(text: "kernel-maintenance: new boot, target release and TUN queue capability verified; rerun commissioning for full sanitation and readiness") + } else { kernel_maintenance_refused(reason: "capability is supported but activation of the reviewed kernel in a new boot is not established") } + } + HostKernelDependencyPlanned { plan: current } => if !kernel_maintenance_plan_continues(reviewed: plan, current: current, action: action) { + kernel_maintenance_refused(reason: "reviewed kernel plan drifted; refuse installation or reboot against a different boot or package proposal") + } else if (action == KernelVerify) { kernel_maintenance_refused(reason: "host still lacks the required queue capability") } + else { kernel_maintenance_apply_observed(plan: plan, current: current, action: action) } + } + } } +} + +fn host_kernel_dependency_drain_cli() -> CliWireResponse { kernel_maintenance_cli(action: KernelDrain) } +fn host_kernel_dependency_install_cli() -> CliWireResponse { kernel_maintenance_cli(action: KernelInstall) } +fn host_kernel_dependency_activate_cli() -> CliWireResponse { kernel_maintenance_cli(action: KernelActivate) } +fn host_kernel_dependency_verify_cli() -> CliWireResponse { kernel_maintenance_cli(action: KernelVerify) } diff --git a/dag/gunbc/recurring_failure_mode/unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider.dag b/dag/gunbc/recurring_failure_mode/unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider.dag index 58d3cf5170a..49ec3d62fd8 100644 --- a/dag/gunbc/recurring_failure_mode/unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider.dag +++ b/dag/gunbc/recurring_failure_mode/unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provider.dag @@ -9,6 +9,7 @@ data unsupported_operation_stops_at_diagnostics_instead_of_demanding_its_provide "2026-10-09, PR #13607: safe workspace reuse selected a TUN descriptor census requiring the kernel attached-queue query, but commissioning carried only its failure text. The earliest missing edge was from the selected operation to its environmental requirement, before a kernel package or reboot could be chosen. The CLI reader also collapsed unsupported and unreadable into one refusal, so it could not safely select a remedy.", "The census now owns TunAttachedQueueRequirement. The read-only Linux ioctl reader preserves numeric EOPNOTSUPP; all other failures stay unreadable. Commissioning consumes that requirement and the existing ensure decision: a supported query needs no provider, including a positive queue count that still fails sanitation; unsupported reaches host-selected Ubuntu package/activation planning. Host/revision/interface/boot and package observations bind the fleet plan. No version comparison establishes capability and the same already-running unsupported target refuses a reboot loop.", "The production commissioning scope carries the derived dependency and the existing AwaitingCapability terminal. srv1 has no admitted reset selection, verified runner drain or console recovery, and this session has no shared-host reboot approval. Those are an explicit maintenance-actuation frontier, not a successful apply. Generic compiler completion described in ensure-closure-design remains unimplemented. Attainable ceiling: the operation demand is shared by construction; external running-kernel support requires live readback. The package/read-error controls and fleet artifact/admission witness exercise production folds; the wet reader witness executes the Python/native ioctl boundary.", + "2026-10-09 continuation: the operator authorized shared-host maintenance. gunbc.host_kernel_dependency_apply now executes the planner’s canonical dependency through root-only drain, exact non-removing installation, normal OS activation and post-boot capability verification. The drain uses runtime masks without --now and the existing descendant cgroup observer. Its first live invocation placed twelve masks and correctly refused while the jobs were still running. BMC reset-subject enrollment is not a prerequisite for a normal OS reboot; it remains a different effect. Generic slot apply still cannot turn a slot request into authority to reboot its shared host.", ], evidence: [], } diff --git a/dag/test/claim/host_kernel_boot_selection_witness_test.dag b/dag/test/claim/host_kernel_boot_selection_witness_test.dag new file mode 100644 index 00000000000..8cb100ec301 --- /dev/null +++ b/dag/test/claim/host_kernel_boot_selection_witness_test.dag @@ -0,0 +1,16 @@ +module test.claim.host_kernel_boot_selection_witness_test + +import std.types { Bool, String } +import gunbc.host_kernel_boot_selection { kernel_maintenance_grub_selects } + +test fn activation_requires_exact_boot_images_and_no_pending_override() -> Bool { + let next_entry_reference = "$\{next_entry}" + let config = join(["if [ \"", next_entry_reference, "\" ] ; then\n set default=\"", next_entry_reference, + "\"\nelse\n set default=\"0\"\nfi\nmenuentry 'Ubuntu' {\n\tlinux\t/vmlinuz-7.0.0-38-generic root=UUID=fixture ro\n\tinitrd\t/initrd.img-7.0.0-38-generic\n}\n"], "") + kernel_maintenance_grub_selects(config: config, environment: "# GRUB Environment Block\n", target: "7.0.0-38-generic") + && !kernel_maintenance_grub_selects(config: config, environment: "next_entry=old-kernel\n", target: "7.0.0-38-generic") + && !kernel_maintenance_grub_selects(config: config, environment: "initrdfail=1\nprev_entry=old\n", target: "7.0.0-38-generic") + && !kernel_maintenance_grub_selects(config: config, environment: "", target: "6.8.0-138-generic") + && !kernel_maintenance_grub_selects(config: concat(config, "set default=\"2\"\n"), environment: "", target: "7.0.0-38-generic") + && !kernel_maintenance_grub_selects(config: "", environment: "", target: "7.0.0-38-generic") +} diff --git a/dag/test/claim/host_kernel_maintenance_witness_test.dag b/dag/test/claim/host_kernel_maintenance_witness_test.dag new file mode 100644 index 00000000000..0097d667180 --- /dev/null +++ b/dag/test/claim/host_kernel_maintenance_witness_test.dag @@ -0,0 +1,81 @@ +module test.claim.host_kernel_maintenance_witness_test + +import std.types { Bool, String, NonEmptyStr, CommitSha } +import extdeps.linux.kernel { LinuxKernelRelease } +import extdeps.linux.tun_fdinfo { tun_attached_queue_requirement } +import gunbc.fleet_host_identity { operator_host_srv1, operator_host_srv2 } +import gunbc.host_kernel_dependency { HostKernelDependencyPlan, UbuntuTunKernelProvider, InstallKernelAndActivate, ActivateInstalledKernel } +import gunbc.host_kernel_dependency_wire { host_kernel_dependency_text } +import gunbc.host_kernel_dependency_apply { + kernel_maintenance_plan_continues, kernel_maintenance_plan_read, kernel_maintenance_runner_units, + kernel_maintenance_unit_readback_admits, KernelDrain, KernelInstall, KernelActivate, +} +import gunbc.runner_slot_retirement { CgroupAbsent, CgroupObservedEmpty, CgroupObservedPopulated, CgroupUnobserved } + +fn maintenance_fixture() -> HostKernelDependencyPlan { + HostKernelDependencyPlan { host: operator_host_srv1, + source_revision: "0123456789abcdef0123456789abcdef01234567" as CommitSha, + requirement: tun_attached_queue_requirement(name: "gunbc-tap13"), + running_release: "6.8.0-138-generic" as LinuxKernelRelease, boot_id: "old-boot", + provider: UbuntuTunKernelProvider { package: "linux-generic-hwe-24.04", version: "7.0.0-38.38~24.04.4", + architecture: "arm64", kernel_release: "7.0.0-38-generic" as LinuxKernelRelease }, + remedy: InstallKernelAndActivate, package_simulation: "reviewed exact package transaction" } +} + +fn maintenance_with_state(plan: HostKernelDependencyPlan, boot: String, installed: Bool, simulation: String) -> HostKernelDependencyPlan { + HostKernelDependencyPlan { host: plan.host, source_revision: plan.source_revision, + requirement: plan.requirement, running_release: plan.running_release, boot_id: boot as NonEmptyStr, + provider: plan.provider, remedy: if installed { ActivateInstalledKernel } else { InstallKernelAndActivate }, + package_simulation: simulation } +} + +test fn installed_readback_can_continue_the_reviewed_plan_without_reinstalling() -> Bool { + let plan = maintenance_fixture() + let after = maintenance_with_state(plan: plan, boot: "old-boot", installed: true, simulation: "") + kernel_maintenance_plan_continues(reviewed: plan, current: after, action: KernelInstall) + && kernel_maintenance_plan_continues(reviewed: plan, current: after, action: KernelActivate) + && !kernel_maintenance_plan_continues(reviewed: plan, current: after, action: KernelDrain) +} + +test fn a_different_boot_or_changed_package_transaction_refuses() -> Bool { + let plan = maintenance_fixture() + !kernel_maintenance_plan_continues(reviewed: plan, + current: maintenance_with_state(plan: plan, boot: "another-boot", installed: true, simulation: ""), action: KernelActivate) + && !kernel_maintenance_plan_continues(reviewed: plan, + current: maintenance_with_state(plan: plan, boot: "old-boot", installed: false, simulation: "different package transaction"), action: KernelInstall) +} + +test fn maintenance_reads_the_real_canonical_dependency_and_refuses_unknown_fields() -> Bool { + let text = host_kernel_dependency_text(plan: maintenance_fixture()) + (match kernel_maintenance_plan_read(text: text) { Present { value: plan } => plan == maintenance_fixture() Absent => false }) + && (match kernel_maintenance_plan_read(text: concat(text, "\n")) { Present { value: _ } => false Absent => true }) + && (match kernel_maintenance_plan_read(text: "{}") { Present { value: _ } => false Absent => true }) +} + +test fn runner_census_includes_loaded_historical_slots_not_only_desired_width() -> Bool { + match kernel_maintenance_runner_units(host: operator_host_srv1, + text: "actions-runner@srv1-01.service loaded active running Runner\nactions-runner@srv1-21.service masked inactive dead Runner\n") { + Present { value: units } => units == ["actions-runner@srv1-01.service", "actions-runner@srv1-21.service"] + Absent => false + } +} + +test fn runner_census_refuses_foreign_empty_duplicate_and_malformed_populations() -> Bool { + all(["", "actions-runner@srv2-01.service loaded active running Runner\n", + "actions.runner.foreign.service loaded active running Runner\n", + "actions-runner@srv1-01.service loaded active running Runner\nactions-runner@srv1-01.service loaded active running Runner\n", + "unreadable\n"], text => match kernel_maintenance_runner_units(host: operator_host_srv1, text: text) { + Present { value: _ } => false Absent => true + }) +} + +test fn drain_requires_inhibition_stopped_service_no_job_and_empty_subtree() -> Bool { + all([CgroupAbsent, CgroupObservedEmpty], occupancy => kernel_maintenance_unit_readback_admits( + state: "inactive", mask: "masked-runtime", job: "", require_mask: true, occupancy: occupancy)) + && all([CgroupObservedPopulated, CgroupUnobserved { cause: "permission denied" }], occupancy => + !kernel_maintenance_unit_readback_admits(state: "inactive", mask: "masked-runtime", job: "", require_mask: true, occupancy: occupancy)) + && all(["active", "activating", "deactivating", "reloading", "unknown", ""], state => + !kernel_maintenance_unit_readback_admits(state: state, mask: "masked-runtime", job: "", require_mask: true, occupancy: CgroupAbsent)) + && !kernel_maintenance_unit_readback_admits(state: "inactive", mask: "enabled", job: "", require_mask: true, occupancy: CgroupAbsent) + && !kernel_maintenance_unit_readback_admits(state: "inactive", mask: "masked", job: "123", require_mask: true, occupancy: CgroupAbsent) +} diff --git a/docs/plans/srv1-workspace-vm-bringup.md b/docs/plans/srv1-workspace-vm-bringup.md index 245b0e748cc..7b82f8de855 100644 --- a/docs/plans/srv1-workspace-vm-bringup.md +++ b/docs/plans/srv1-workspace-vm-bringup.md @@ -419,7 +419,7 @@ to multi-queue would not match [Firecracker v1.16.1's open flags](https://github No refusal has been relaxed, no readiness record has been fabricated, and no VM has been commissioned or booted. -### Proposed host maintenance (not applied) +### Host kernel dependency and maintenance The commissioning planner derives this proposal from the selected sanitation reader's `TunAttachedQueueRequirement`. Its observation now invokes Linux's @@ -441,14 +441,42 @@ refuses instead of entering a reboot loop. The existing `workspace-slot-commissioning` plan carries the dependency, package simulation and continuation. It cannot mint a commissioning credential or initial -readiness. Its apply admission is `AwaitingCapability`: plan-bound interruption -authority, runner drain, verified console recovery and an external boot observer -are not established for srv1. `gunbc.host_reset_boot_selection` currently admits -only the separately enrolled reset subject, not srv1. This change implements -capability-driven dependency planning; it does **not** implement an unattended -srv1 kernel installer/reboot actuator. After an admitted maintenance path establishes -the running capability, rerunning the same commissioning goal continues through -the full sanitation, protected-state and runtime checks. +readiness. Generic slot apply remains `AwaitingCapability` because a slot request +alone does not authorize interruption of the shared host. + +`gunbc.host_kernel_dependency_apply` now supplies the administrator maintenance +path for that exact canonical dependency. Its drain entry puts runtime masks on +observed runner instances without stopping them, allowing their one current job +to finish. Installation and activation require every instance inhibited, stopped, +without a queued job, and with an empty or absent workload cgroup. The existing +cgroup observer includes descendants. Unknown and foreign runner census rows, +unreadable state, a live workspace controller, changed boot identity or changed +package proposal refuse. Existing permanent masks are retained. + +The install entry uses the same exact, non-removing apt selection as planning, +then reads package identities, boot image, initramfs and the TUN symbol back. An +already installed target needs only activation. The separate activation entry +checks srv1's observed GRUB default-zero realization, exact image/initramfs and +absence of a pending boot override, then requests a normal OS reboot. The peer +keeps the reviewed plan and invokes verification after SSH returns. Verification +requires a new boot identity, the target release and the actual queue capability; +commissioning must subsequently perform its full sanitation and readiness checks. + +These entries require root and are not included in the job-user commissioning +sudo grant. They are the operator's explicitly approved maintenance action, not +an unattended reboot policy. The BMC is the operator recovery fallback. Normal OS +reboot does not require enrolling srv1 in the separate BMC-issued reset workflow +(`host_reset_boot_selection`); treating that unrelated enrollment as a kernel +installation dependency was an overly broad frontier in the earlier proposal. + +The four entries consume the planner's canonical JSON on stdin: +`host_kernel_dependency_drain_cli`, `host_kernel_dependency_install_cli`, +`host_kernel_dependency_activate_cli`, and `host_kernel_dependency_verify_cli`. +An unfinished drain reports refusal and leaves its runtime masks in place; it +never interrupts current jobs to force completion. Those masks disappear at +reboot. If maintenance is abandoned before reboot, the operator removes only the +runtime masks created by that attempt and starts those instances again, preserving +previous permanent masks. The pre-maintenance unit census is retained as evidence. This is ordinary domain composition using the existing `ensure` decision and fleet plan, not a claim that the generic completion/elaboration engine proposed @@ -464,10 +492,13 @@ The modules package's `System.map-7.0.0-38-generic` contains `tun_get_channels`, not merely its version. The install command whose simulation was reviewed is: ```sh -sudo apt-get --no-install-recommends install linux-generic-hwe-24.04=7.0.0-38.38~24.04.4 +sudo env LC_ALL=C DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l \ + apt-get --yes --no-install-recommends --no-remove install \ + linux-generic-hwe-24.04=7.0.0-38.38~24.04.4 ``` -This requires separate operator approval for shared-host maintenance: the reboot +The operator approved this shared-host maintenance on 2026-10-09 after the +interruption was explained. The reboot interrupts srv1's twelve runner services, dashboard, fabric storage, approval broker, ntfy and container services. Before installing or rebooting, drain work, establish console/boot recovery, and recheck the package plan and disk capacity. @@ -475,11 +506,29 @@ GRUB currently boots entry 0 with a hidden, zero-second menu; keeping an old kernel alone is not automatic rollback. After reboot, require successful `ethtool --show-channels gunbc-tap13`, verify KVM, network and service recovery, then rerun the commissioning plan. A newer version string alone is insufficient. -Local read-only IPMI device/chassis queries succeed and SOL reports enabled; -remote console access and boot recovery have not been established. The current -GitHub credential cannot list organization runners (403), so a runner drain must -be established separately rather than inferred from the repository runner list. -No kernel package has been installed and no reboot has been initiated. +Local read-only IPMI device/chassis queries succeed and SOL reports enabled. +The BMC's HTTPS endpoint is reachable from srv2, firmware selects Ubuntu, and the +BMC reports no boot-device override. An authenticated remote console session has +not been tested. Runner drain uses the actual local service/cgroup population, +not an incomplete GitHub organization runner listing. + +The first production drain used source snapshot SHA-256 +`d2ef5ab2f7d3d11bda59d5773ffd9ca2a41c8f14578d41746f6ced41da544646` +and the existing canonical dependency for installed release `4fa60f71342`. +It placed twelve runtime masks and returned the expected awaiting-jobs refusal; +the existing jobs remained active and permanent masks were unchanged. Evidence: +`target/srv1-bringup-evidence/kernel-maintenance-drain-r1.log` and the before/masked +runner censuses. Package installation and reboot remain pending at this receipt. + +The seven maintenance controls pass: canonical input, permitted install progress, +boot/transaction drift, runner population inclusion/refusal, complete drain +readback, and exact GRUB image selection with override refusal. The boot-selection +control resolves only its pure reader and passes at 1,216 evaluation steps. +All 8,291 source files parse; formatting and whitespace checks pass. Logs are +`target/srv1-kernel-maintenance-tests-r3.log`, +`target/srv1-kernel-maintenance-drain-tests.log`, +`target/srv1-kernel-maintenance-boot-tests-leaf-r2.log`, and +`target/srv1-kernel-maintenance-parse-r2.log`. The diagnostics repair at `0fd9f91c7b1` passes all 29 focused commissioning, network-producer, TUN-census and initial-sanitation witnesses in