diff --git a/.agents/skills/manage-ci/references/current-inventory.md b/.agents/skills/manage-ci/references/current-inventory.md index b2ac053229..377537a190 100644 --- a/.agents/skills/manage-ci/references/current-inventory.md +++ b/.agents/skills/manage-ci/references/current-inventory.md @@ -86,6 +86,246 @@ workflows remain fixed to GitHub-hosted runners; the PR entrypoints pass no repository secrets. The trusted main entrypoint may pass the optional `HF_TOKEN` for public-fixture rate-limit resilience. +## Prebuilt runner-image containerization + +Some CI jobs run inside a `container:` pinned to a digest from the +`mesh-llm-runner-images` repo instead of installing tooling per-run with +`actions/setup-*`. There is no separate sister repo: the `public web` +backend (baked Chromium/Playwright) lives on `mesh-llm-runner-images` main +alongside every other family, added by `17283ab` (#20, the `public web` +backend) and `5ea673b` (#21, the Playwright version assert). The GHCR +package name +`ghcr.io/mesh-llm/mesh-llm-cuda-runner` is legacy: it hosts every backend +family (`public cpu`, `public cuda`, `public rocm`, `public vulkan`, +`public web`, `self-hosted`), not only CUDA. Each image bakes +`cargo cmake docker git jq just lld node ninja npm pnpm python rustc sccache` +(asserted by `verify-runner-image`, see below) plus a Python venv on `PATH` +(`VIRTUAL_ENV=/opt/mesh-llm/venv`), pinned pnpm/node (`PNPM_HOME`, +`CARGO_HOME`, `RUSTUP_HOME` baked as ENV so they resolve the same regardless +of the container's `HOME`), and, for the `public` stage only, runs as +**root** (`USER root`, never dropped back) rather than `runner` -- +`self-hosted` is the only stage that ends `USER runner`. + +Reusable slices/workflows with a `container:` job, and what backs it: + +| Workflow | Job(s) | Image family | +| --- | --- | --- | +| `ci-{linux}-host-slice.yml`, `ci-linux-runtime-slice.yml`, `ci-linux-product-slice.yml`, `ci-rust-tests-slice.yml`, `ci-quality-slice.yml` (Clippy batches) | matrix-selected | `public cpu` (pre-existing, predates this containerization pass) | +| `native-sdk-artifact.yml`, `node-sdk-addon-artifact.yml`, `static-abi-artifact.yml`, `swift-sdk-artifact.yml` | producer job | `public cpu` (pre-existing) | +| `hf-download-smoke.yml`, `scripted-binary-smoke.yml` | their single job | `public cpu`, sha256:8d93de6b... -- unconditional, no bare-metal row | +| `smoke.yml` | `smoke_tests` | `public cpu` when `inputs.runner != 'gpu-nvidia'`, else uncontainerized (see opt-out below) | +| `sdk-smoke.yml` | its job | `public cpu` when `inputs.sdk_kind != 'swift'`, else uncontainerized | +| `ci-ui-artifact-slice.yml` | `ui_artifact` | `public web`, sha256:1c73f0f2... | +| `ci-web-slice.yml` | `ui_quality`, `ui_e2e`, `website` | `public web` | +| `website-pages.yml` | `build` | `public web` | +| `nightly-stability-run.yml` | `stability` | `public web` (bakes node/pnpm the CLI-smoke step needs) | +| `release.yml` (several CUDA/ROCm/Vulkan build/compose rows) | per-backend `public` digests | pre-existing, unrelated to this containerization work; each row pins its own backend digest via `ci/slices.yml` / job matrix, not a shared convention | + +`public cpu` and `public web` are separate image builds (the latter adds +`PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright`, +`PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1`, and a stamped +`/etc/mesh-runner-playwright-version`); do not assume one digest covers both. + +### `image: ''` opt-out rows + +`smoke.yml`'s `gpu-nvidia` row (the approved uncredentialed self-hosted CUDA +smoke exception) and `sdk-smoke.yml`'s `swift` row (host-only macOS SDK +build, `macos-15`, never container-capable) opt out per-run with +`image: ''` rather than being a separate job, so the rest of the job body +(steps, `if: job.container.id == ''` gates) stays shared. This is proven to +actually opt a job out of containerization by two runs on the temporary +branch-head harness used to validate #1380 (run 32349670919, jobs +96370649138 `CUDA inference smoke` and 96375145155 `swift SDK Smoke`): no +`Initialize containers` log group, no `docker create`, and the gated +`actions/setup-python`/`pnpm/action-setup` steps ran. There is no other +empty-image job anywhere in this repo's workflow history. + +The ternary that selects `image: ''` must put the **non-empty** value in the +`&&` branch: `cond && url || ''`, never `cond && '' || url`. GitHub Actions +expressions are JS-style short-circuit and `''` is falsy, so +`cond && '' || url` always evaluates to `url` regardless of `cond` -- the +opt-out branch becomes unreachable. `scripts/tests/test_ci_workflow_ternary_contract.py` +fails any `${{ }}` ternary whose `&&` branch is a falsy literal (`''`, `""`, +`0`, `false`) across every workflow; it exists specifically because this bug +class is invisible to `actionlint`. + +### `job.container.id == ''` gating + +When a job has both a containerized and a bare-metal row (the two rows +above), `actions/setup-python`, `actions/setup-node`, and +`pnpm/action-setup` steps are gated `if: job.container.id == ''` rather than +deleted, because the bare-metal row still needs them -- the image is not +present there. **Deliberate exception: `actions/setup-java` in +`sdk-smoke.yml` is never gated.** `verify-runner-image`'s asserted tool list +has no JDK, so the image provides nothing for it to shadow; gating it would +break the Kotlin SDK smoke on the containerized row instead of protecting it. +Jobs with no bare-metal row at all (the `ci-web-slice.yml` / `website-pages.yml` +/ `ci-ui-artifact-slice.yml` / `nightly-stability-run.yml` set) delete the +now-redundant setup actions outright instead of gating them -- there is +nothing for the `if:` to select between. + +`npm install --global openai` in `smoke.yml` is **not** gated on +`job.container.id`, even though `install-core-tools.sh:83` bakes an +exact-pinned `openai` into the image on `mesh-llm-runner-images` main. The +`public cpu` digest pinned in `ci/slices.yml` predates that bake (see +"A pinned digest is a frozen artifact" below), so the containerized row needs +the install too, and the step runs unconditionally for both it and the +bare-metal `gpu-nvidia` row. Re-gate it only once the CPU digest is promoted +past `mesh-llm-runner-images` #20 and that is confirmed from a green run. + +Both call sites install `openai@7.5.0`, not floating `openai`. The step runs +with the full job environment (`HF_TOKEN` included) and npm lifecycle scripts +inherit it, so an unreviewed upstream release must not be able to execute +there; `zizmor`'s `adhoc-packages` rule flags the floating form. The version +deliberately tracks the image's own `ARG OPENAI_NPM_VERSION` +(`mesh-llm-runner-images` `Dockerfile:25`) so that re-gating the step later is +a no-op rather than a version swap -- bump both sides together. + +### Container jobs default `run:` to `sh`, not `bash` + +Jobs with a `container:` block resolve the default `run:` shell to +`sh -e {0}`, not `bash -e {0}` (bare-metal Linux/macOS runners default to +bash; this only changes inside a container). Composite actions are +unaffected -- they declare their own shell. Any `run:` step in a +containerized job that uses a bashism (`<<<`, `set -o pipefail`, `[[`, +array assignment, `${v//}`/`${v^^}`/`${v,,}`, `&>`, `source`, `+=(`, ...) +must declare `shell: bash` explicitly or it fails at runtime with a +`dash`/`sh` syntax error that `actionlint` cannot catch -- its shellcheck +integration assumes bash. Two sites hit this in the same PR: +`ci-web-slice.yml`'s `ui_e2e` preflight (`<<<`) and +`website-pages.yml`'s `Stage Pages artifact` (`set -euo pipefail`); both now +declare `shell: bash`. + +`$(( ))` arithmetic expansion is **not** on that list and must not be added. +It is POSIX (Shell Command Language 2.6.4) and `dash` evaluates it correctly; +flagging it would reject valid `sh` steps and force a spurious `shell: bash`. +`scripts/tests/test_ci_workflow_container_shell_contract.py` carries the +pattern list and an inline note saying so. + +### Reusable-workflow permission chain + +A called reusable workflow may not request a permission scope its caller job +does not grant; GitHub rejects at run creation with a **zero-job +`startup_failure`** -- no jobs, no logs, no check run on the commit, and +`actionlint` cannot see it. Containerizing surfaced this because +`packages: read` (needed to pull the private GHCR runner images) has to be +granted at *every* hop, and +`ci-linux-product-smoke-slice.yml` / `ci-macos-product-smoke-slice.yml` sat at +`contents: read` between granted parents and requesting children. +`scripts/tests/test_ci_workflow_permission_contract.py` walks every local +`uses: ./.github/workflows/X.yml` edge and asserts the caller's effective +permissions (job-level, else workflow-level) cover what `X.yml` requests. + +Two properties make that assertion real rather than decorative, and both were +absent when the test was first written: + +1. **The callee's requested set is the workflow-level block merged with every + explicit job-level block.** Five reusable workflows here + (`native-sdk-artifact.yml`, `node-sdk-addon-artifact.yml`, `sdk-smoke.yml`, + `static-abi-artifact.yml`, `swift-sdk-artifact.yml`) declare permissions + only at job level, so reading the workflow-level block alone returns `None` + for them and skips their caller edges entirely -- including the + `packages: read` edges this test exists to cover. +2. **Scope levels are compared, not scope names.** `contents: read` does not + satisfy a callee's `contents: write`; GitHub rejects that downgrade at run + creation exactly like a missing scope. The comparison ranks + `none < read < write`, and where a scope is declared in more than one block + the strictest level wins. A name-only set comparison silently passes the + downgrade. + +3. **`read-all`/`write-all` are modelled on the granting side, not skipped.** + As a *grant* they are perfectly enumerable -- `write-all` satisfies any + request, `read-all` satisfies a `read` request but not a `write` one -- so + returning "unknown" and skipping the edge would hide the same + run-creation failure. As a *request* they stay opaque: a callee asking + `write-all` names no scopes to hold its caller to, and asserting there + would be invention rather than checking. +4. **Both workflow extensions are read.** Globbing `*.yml` alone would skip a + `*.yaml` callee entirely; the repo has none today, which is exactly when + that gap is cheapest to close. + +None of these were breakages -- the repo satisfies the contract at every edge +under the strict check, and it has no `.yaml` workflows or all-scope grants at +all. That is the point: a permission test that under-reads its inputs reports +green for edges it never examined, and each of these was found by tightening +the test rather than by anything failing. + +### `verify-runner-image` preflight + +Containerized jobs run `verify-runner-image ...` +(positional args: environment, backend, mesh-llm revision, CUDA series, ROCm +version, runner-images revision, and -- `public web` only -- expected +Playwright version, added in `mesh-llm-runner-images`#21) before doing real +work, asserting `/etc/mesh-runner-*` files match what the job expects rather +than trusting the digest pin alone. `ci-web-slice.yml`'s `ui_e2e` job +resolves the installed `@playwright/test` version with +`pnpm exec playwright --version | head -n1 | awk '{print $NF}'` (guarded by a +`^[0-9]+\.[0-9]+\.[0-9]+$` shape assertion -- `playwright --version` can share +stdout with an npm warning) and passes it as the seventh argument; a mismatch +against the image's own build-time `playwright --version` fails fast instead +of surfacing as a confusing Playwright/Chromium error deep in the E2E run. + +`crates/mesh-llm-ui/package.json`'s `@playwright/test` and +`mesh-llm-runner-images`' `config/playwright-pin.txt` are now a matched pair +(both `1.62.1` as of 2026-08-20; re-check the two sources rather than +trusting this line). Bumping the mesh-llm side alone fails `ui_e2e` on +**every** PR at this preflight, not just locally. The bump is a four-step +cross-repo sequence, in order: bump `config/playwright-pin.txt` in +`mesh-llm-runner-images`, rebuild and promote the `public web` image, re-pin +the new digest in `ci-web-slice.yml` (and `ci-ui-artifact-slice.yml` / +`website-pages.yml` / `nightly-stability-run.yml`, which share it), then +bump `@playwright/test` in `crates/mesh-llm-ui/package.json`. + +### `setup-macos-lld` composite + +`.github/actions/setup-macos-lld` replaces per-callsite +`brew install lld` plus a hand-rolled `PATH`/`RUSTFLAGS` export with one +composite: install lld via brew, resolve `$(brew --prefix lld)/bin`, link a +real `edition = "2024"` probe binary with `-Clink-arg=-fuse-ld=lld`, then +export `CARGO_ENCODED_RUSTFLAGS=-Clink-arg=-fuse-ld=lld` and the resolved bin +directory. `CARGO_ENCODED_RUSTFLAGS` **replaces** any +`target..rustflags` from a checked-in `.cargo/config.toml` rather +than merging with it -- confirmed safe here because every call site is +macOS-gated and no call site touches an `android` target (the repo's +`.cargo/config.toml` android entries carry a `-Wl,-z,max-page-size=16384` +flag that would otherwise silently stop applying). Seven call sites: +`ci-platform-checks-slice.yml`, `ci-macos-host-slice.yml`, +`swift-sdk-artifact.yml`, `native-sdk-artifact.yml`, +`node-sdk-addon-artifact.yml`, and two in `release.yml`. Only the first +three were reachable by the temporary branch-head harness used to validate +PR #1380 (no macOS row in +`native-sdk-artifact.yml`/`node-sdk-addon-artifact.yml` ran there, and +`release.yml` only runs on an actual release cut) -- the other four are +statically cleared (macOS-gated, no android target on any of them) rather +than proven by a real run. `node-sdk-addon-artifact.yml`'s own +`Validate macOS x64 cross-linker` step is a deliberate near-duplicate of the +composite's probe, not dead code: it passes `--target x86_64-apple-darwin` +where the composite only probes the host target. + +### A pinned digest is a frozen artifact + +`mesh-llm-runner-images` HEAD says nothing about what is inside the digest a +workflow pins -- the `public cpu` digest pinned in `ci/slices.yml` was built +2026-07-22 and does not contain changes merged to that repo afterwards +(the `smoke.yml` openai bake landed a week later, in #20). Before deleting or +gating a dependency install on the grounds that "the image bakes it," +confirm the capability exists **in the pinned digest**, and confirm it from a +green run of the job that needs it. `verify-runner-image`'s JSON is the +cheap probe: `mesh_llm_revision` dates the build, and missing keys (added to +the asserted object in later `mesh-llm-runner-images` commits) date the +baked verify script itself. + +### Digest promotion + +`build-and-push.yml` (in `mesh-llm-runner-images`) runs `stage_families` for +both `operation=stage` and `operation=promote`; `promote_versioned` reads the +candidate descriptor artifact from that **same run**, not from an earlier +stage run. A `promote` dispatch therefore re-stages and promotes its own +build. Read the digest to pin from the promote job's own `digest=` output +(e.g. `promoted ghcr.io/... -> sha256:...` in its log) -- never carry forward +a digest observed from an earlier stage-only run, even one at the same +source commit. + ## Planner contract - `scripts/plan-ci.py` is the only routing implementation. diff --git a/.github/actions/setup-macos-lld/action.yml b/.github/actions/setup-macos-lld/action.yml new file mode 100644 index 0000000000..bf37860a37 --- /dev/null +++ b/.github/actions/setup-macos-lld/action.yml @@ -0,0 +1,38 @@ +name: Set up macOS lld linker +description: > + Install and verify Homebrew's lld on a macOS runner. Always probe-links a + throwaway crate before any caller trusts the result -- a bad install must + fail loudly, not build silently against the wrong (or no) linker. + Deliberately uncached: mesh-llm's Actions cache is already over the 10GB + per-repo quota (see gh cache list), so a ~1.9GB lld/llvm/z3 Cellar entry + here would evict the far more expensive macOS Rust target/ cache for a + 15-30s bottle-pour saving. Revisit only with a measured number and a + quota plan, not an assumption. + +runs: + using: composite + steps: + - name: Install lld + shell: bash + run: brew install lld + + - name: Verify lld links a real binary + shell: bash + run: | + set -euo pipefail + lld_bin="$(brew --prefix lld)/bin" + test -x "$lld_bin/ld64.lld" + probe="$RUNNER_TEMP/mesh-setup-macos-lld-probe" + mkdir -p "$probe/src" + printf '[package]\nname = "mesh-setup-macos-lld-probe"\nversion = "0.0.0"\nedition = "2024"\n' > "$probe/Cargo.toml" + printf 'fn main() {}\n' > "$probe/src/main.rs" + PATH="$lld_bin:$PATH" CARGO_ENCODED_RUSTFLAGS='-Clink-arg=-fuse-ld=lld' \ + cargo rustc --quiet --manifest-path "$probe/Cargo.toml" + echo "$lld_bin" >> "$GITHUB_PATH" + # Job-global via GITHUB_ENV, and cargo prefers CARGO_ENCODED_RUSTFLAGS + # over RUSTFLAGS -- any later step in the same job that sets RUSTFLAGS + # is silently ignored, and any macOS job that cross-builds an android + # target loses .cargo/config.toml's -Wl,-z,max-page-size=16384 for + # that job. Verified safe today: no other RUSTFLAGS setter and no + # android targets on any of this action's macOS call sites. + echo 'CARGO_ENCODED_RUSTFLAGS=-Clink-arg=-fuse-ld=lld' >> "$GITHUB_ENV" diff --git a/.github/workflows/ci-linux-host-slice.yml b/.github/workflows/ci-linux-host-slice.yml index 31383c6952..530be31a8a 100644 --- a/.github/workflows/ci-linux-host-slice.yml +++ b/.github/workflows/ci-linux-host-slice.yml @@ -120,7 +120,7 @@ jobs: ref: ${{ inputs.source_sha || github.sha }} persist-credentials: false - name: Verify prebuilt host environment - run: verify-runner-image public + run: verify-runner-image public cpu - uses: ./.github/actions/configure-sccache-gha with: allow_depot_remote_cache: ${{ needs.runner_policy.outputs.allow_depot_remote_cache }} diff --git a/.github/workflows/ci-linux-product-slice.yml b/.github/workflows/ci-linux-product-slice.yml index e312ed97cd..f72a52f4d5 100644 --- a/.github/workflows/ci-linux-product-slice.yml +++ b/.github/workflows/ci-linux-product-slice.yml @@ -98,7 +98,7 @@ jobs: ref: ${{ inputs.source_sha || github.sha }} persist-credentials: false - name: Verify product composition environment - run: verify-runner-image public + run: verify-runner-image public cpu - uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0 with: name: ci-host-linux-${{ matrix.runtime.architecture }} diff --git a/.github/workflows/ci-linux-product-smoke-slice.yml b/.github/workflows/ci-linux-product-smoke-slice.yml index 6a02a21ee6..f8f633f7a8 100644 --- a/.github/workflows/ci-linux-product-smoke-slice.yml +++ b/.github/workflows/ci-linux-product-smoke-slice.yml @@ -26,6 +26,7 @@ on: permissions: contents: read + packages: read jobs: core: diff --git a/.github/workflows/ci-macos-host-slice.yml b/.github/workflows/ci-macos-host-slice.yml index 3f3d4151a8..f739175a5d 100644 --- a/.github/workflows/ci-macos-host-slice.yml +++ b/.github/workflows/ci-macos-host-slice.yml @@ -115,7 +115,7 @@ jobs: shared-key: ci-host-macos-${{ inputs.profile }} save-if: ${{ github.ref == 'refs/heads/main' && github.event.inputs.original_event_name != 'pull_request' && github.event.inputs.original_event_name != 'pull_request_target' }} - name: Install host linker - run: brew install lld + uses: ./.github/actions/setup-macos-lld - name: Download immutable UI distribution uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0 with: diff --git a/.github/workflows/ci-macos-product-smoke-slice.yml b/.github/workflows/ci-macos-product-smoke-slice.yml index 08e12536de..57dc5b7edb 100644 --- a/.github/workflows/ci-macos-product-smoke-slice.yml +++ b/.github/workflows/ci-macos-product-smoke-slice.yml @@ -30,6 +30,7 @@ on: permissions: contents: read + packages: read jobs: metal_model_load: diff --git a/.github/workflows/ci-macos-runtime-slice.yml b/.github/workflows/ci-macos-runtime-slice.yml index 1bed4e683f..5e50f0a30d 100644 --- a/.github/workflows/ci-macos-runtime-slice.yml +++ b/.github/workflows/ci-macos-runtime-slice.yml @@ -98,8 +98,6 @@ jobs: with: ref: ${{ inputs.source_sha || github.sha }} persist-credentials: false - - name: Install Metal runtime dependencies - run: brew install cmake ninja jq - uses: ./.github/actions/resolve-native-toolchain-epoch with: include_tool_versions: "true" diff --git a/.github/workflows/ci-platform-checks-slice.yml b/.github/workflows/ci-platform-checks-slice.yml index deab54ea6e..8667f8c94c 100644 --- a/.github/workflows/ci-platform-checks-slice.yml +++ b/.github/workflows/ci-platform-checks-slice.yml @@ -114,14 +114,7 @@ jobs: - uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 # stable 2026-07-16 - name: Configure macOS linker if: ${{ matrix.check.platform == 'macos' }} - run: | - brew install lld - mkdir -p .cargo - lld_prefix="$(brew --prefix lld)" - cat > .cargo/config.toml <&2; exit 1; } + verify-runner-image public web "" "" "" "" "$resolved" - name: Run console E2E run: pnpm run test:e2e - name: Upload E2E failure artifacts @@ -193,6 +182,8 @@ jobs: if: ${{ inputs.website_changed }} runs-on: ${{ needs.runner_policy.outputs.runner_4 }} timeout-minutes: 20 + container: + image: ghcr.io/mesh-llm/mesh-llm-cuda-runner@sha256:1c73f0f2e4b2ce657cc9dc49c9c27374f3c364af19f884f1970331c3689da5d2 steps: - uses: Mesh-LLM/mesh-llm/.github/actions/audit-depot-pr-isolation@ed07043b84d720aab30e75ed2f038f7042576f16 with: @@ -204,15 +195,6 @@ jobs: with: ref: ${{ inputs.source_sha || github.sha }} persist-credentials: false - - uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0 - with: - node-version: 24 - package-manager-cache: ${{ needs.runner_policy.outputs.allow_native_github_cache == 'true' }} - cache: ${{ needs.runner_policy.outputs.allow_native_github_cache == 'true' && 'npm' || '' }} - cache-dependency-path: | - .github/cache-version.txt - website/package-lock.json - - uses: taiki-e/install-action@3d23c1bbdafe696dfccad2664945a04f47d03dc3 # just - name: Install website dependencies working-directory: website run: npm ci diff --git a/.github/workflows/ci-website-lane.yml b/.github/workflows/ci-website-lane.yml index 8442fd9a04..65164f4037 100644 --- a/.github/workflows/ci-website-lane.yml +++ b/.github/workflows/ci-website-lane.yml @@ -40,6 +40,7 @@ on: inputs: *lane_inputs permissions: contents: read + packages: read concurrency: group: ci-website-${{ inputs.supersession_key || inputs.source_sha }} diff --git a/.github/workflows/hf-download-smoke.yml b/.github/workflows/hf-download-smoke.yml index 5439ae09b3..5ecae89ea8 100644 --- a/.github/workflows/hf-download-smoke.yml +++ b/.github/workflows/hf-download-smoke.yml @@ -22,14 +22,21 @@ env: permissions: contents: read + packages: read jobs: hf_download_smoke: name: HuggingFace download smoke - # This credential-bearing reusable workflow is intentionally ineligible for - # Depot and self-hosted runner groups. + # Model credentials stay on an isolated GitHub-hosted runner; the + # container just supplies prebuilt smoke utilities, it does not move + # execution to self-hosted/depot infra. runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.timeout_minutes }} + container: + image: ghcr.io/mesh-llm/mesh-llm-cuda-runner@sha256:8d93de6ba30173e825a16fdecf011f9c632edc6e1259df7289e491b0a05f829d + defaults: + run: + shell: bash env: HF_TOKEN: ${{ secrets.HF_TOKEN }} HUGGING_FACE_HUB_TOKEN: ${{ secrets.HF_TOKEN }} @@ -39,11 +46,16 @@ jobs: ref: ${{ inputs.source_sha || github.sha }} persist-credentials: false - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0 + # The prebuilt image bakes a venv on PATH; a host-side setup-python + # shadows it, leaving a bare interpreter with none of the image's + # installed deps. This job always runs in a container, so this step + # is now always skipped, not just conditionally redundant. + if: job.container.id == '' with: python-version: "3.12" - uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 # stable 2026-07-16 - - name: Install system dependencies - run: sudo apt-get update && sudo apt-get install -y pkg-config libssl-dev libdbus-1-dev lld + - name: Verify prebuilt smoke environment + run: verify-runner-image public cpu - name: Configure Linux Rust linker run: | mkdir -p .cargo diff --git a/.github/workflows/main_website.yml b/.github/workflows/main_website.yml index 7ae7ddc9d6..8de54a7130 100644 --- a/.github/workflows/main_website.yml +++ b/.github/workflows/main_website.yml @@ -69,6 +69,7 @@ jobs: permissions: checks: write contents: read + packages: read uses: ./.github/workflows/ci-website-lane.yml with: lane_plan_json: ${{ needs.plan.outputs.lane_plan }} diff --git a/.github/workflows/native-sdk-artifact.yml b/.github/workflows/native-sdk-artifact.yml index 7cf674afa9..cd81314c67 100644 --- a/.github/workflows/native-sdk-artifact.yml +++ b/.github/workflows/native-sdk-artifact.yml @@ -250,7 +250,7 @@ jobs: fi - name: Verify prebuilt native SDK environment - run: verify-runner-image public + run: verify-runner-image public cpu - name: Resolve static ABI toolchain epoch uses: ./.github/actions/resolve-native-toolchain-epoch @@ -357,7 +357,9 @@ jobs: allow_native_github_cache: ${{ needs.runner_policy.outputs.allow_native_github_cache }} - name: Install macOS native SDK dependencies - run: brew install cmake ninja lld + run: brew install cmake ninja + - name: Install macOS native SDK linker + uses: ./.github/actions/setup-macos-lld - name: Resolve macOS native toolchain epoch uses: ./.github/actions/resolve-native-toolchain-epoch diff --git a/.github/workflows/nightly-stability-run.yml b/.github/workflows/nightly-stability-run.yml index 8e9eb41c16..2b1cfdbd29 100644 --- a/.github/workflows/nightly-stability-run.yml +++ b/.github/workflows/nightly-stability-run.yml @@ -34,6 +34,7 @@ on: permissions: contents: read + packages: read jobs: stability: @@ -42,6 +43,8 @@ jobs: # caller-selected label for a privileged runner group. runs-on: ubuntu-24.04 timeout-minutes: 120 + container: + image: ghcr.io/mesh-llm/mesh-llm-cuda-runner@sha256:1c73f0f2e4b2ce657cc9dc49c9c27374f3c364af19f884f1970331c3689da5d2 env: MESH_STABILITY_BASE_URL: ${{ inputs.base_url }} MESH_STABILITY_MODELS: ${{ inputs.models }} @@ -80,18 +83,13 @@ jobs: echo "run=true" >> "$GITHUB_OUTPUT" - - uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0 - if: ${{ steps.preflight.outputs.run == 'true' && inputs.agent_smokes != '' }} - with: - node-version: 24 - - name: Install requested agent CLIs + # node/pnpm come from the prebuilt image; corepack is unneeded since + # the image already bakes the pinned pnpm version on PATH. if: ${{ steps.preflight.outputs.run == 'true' && inputs.agent_smokes != '' }} shell: bash run: | set -euo pipefail - corepack enable - corepack prepare pnpm@10 --activate export PNPM_HOME="$HOME/.local/share/pnpm" mkdir -p "$PNPM_HOME" echo "PNPM_HOME=$PNPM_HOME" >> "$GITHUB_ENV" diff --git a/.github/workflows/nightly-stability.yml b/.github/workflows/nightly-stability.yml index 9a53742096..3db2abd640 100644 --- a/.github/workflows/nightly-stability.yml +++ b/.github/workflows/nightly-stability.yml @@ -36,6 +36,7 @@ on: permissions: contents: read + packages: read concurrency: group: nightly-stability diff --git a/.github/workflows/node-sdk-addon-artifact.yml b/.github/workflows/node-sdk-addon-artifact.yml index bba2ecfe24..5c9fd83f3f 100644 --- a/.github/workflows/node-sdk-addon-artifact.yml +++ b/.github/workflows/node-sdk-addon-artifact.yml @@ -194,15 +194,14 @@ jobs: allow_depot_remote_cache: "false" allow_native_github_cache: "true" - name: Install macOS native dependencies - run: brew install cmake ninja lld - - name: Configure and validate macOS x64 linker + run: brew install cmake ninja + - name: Install macOS native linker + uses: ./.github/actions/setup-macos-lld + - name: Validate macOS x64 cross-linker if: ${{ inputs.target == 'darwin-x64' }} run: | set -euo pipefail lld_bin="$(brew --prefix lld)/bin" - test -x "$lld_bin/ld64.lld" - echo "$lld_bin" >> "$GITHUB_PATH" - echo 'CARGO_ENCODED_RUSTFLAGS=-Clink-arg=-fuse-ld=lld' >> "$GITHUB_ENV" probe="$RUNNER_TEMP/mesh-cargo-linker-check" mkdir -p "$probe/src" printf '[package]\nname = "mesh-cargo-linker-check"\nversion = "0.0.0"\nedition = "2024"\n' > "$probe/Cargo.toml" diff --git a/.github/workflows/pr_website.yml b/.github/workflows/pr_website.yml index 8a9158cb6f..a5e76afcef 100644 --- a/.github/workflows/pr_website.yml +++ b/.github/workflows/pr_website.yml @@ -67,6 +67,7 @@ jobs: permissions: checks: write contents: read + packages: read uses: Mesh-LLM/mesh-llm/.github/workflows/ci-website-lane.yml@main with: lane_plan_json: ${{ needs.plan.outputs.lane_plan }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a2ca692528..c6cb7f0433 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -263,7 +263,11 @@ jobs: - name: Install macOS dependencies if: runner.os == 'macOS' - run: brew install cmake ninja lld + run: brew install cmake ninja + + - name: Install macOS linker + if: runner.os == 'macOS' + uses: ./.github/actions/setup-macos-lld - name: Prepare dispatched release version if: github.event_name == 'workflow_dispatch' @@ -509,7 +513,12 @@ jobs: - name: Install macOS dependencies if: runner.os == 'macOS' - run: brew install cmake ninja lld + run: brew install cmake ninja + + - name: Install macOS linker + if: runner.os == 'macOS' + uses: ./.github/actions/setup-macos-lld + - name: Resolve hosted native toolchain epoch uses: ./.github/actions/resolve-native-toolchain-epoch with: diff --git a/.github/workflows/scripted-binary-smoke.yml b/.github/workflows/scripted-binary-smoke.yml index 240e367a96..3c171dbf57 100644 --- a/.github/workflows/scripted-binary-smoke.yml +++ b/.github/workflows/scripted-binary-smoke.yml @@ -55,13 +55,21 @@ env: permissions: contents: read + packages: read jobs: scripted_binary_smoke: name: Scripted Binary Smoke - # Model credentials stay on an isolated GitHub-hosted runner. + # Model credentials stay on an isolated GitHub-hosted runner; the + # container just supplies prebuilt smoke utilities, it does not move + # execution to self-hosted/depot infra. runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.timeout_minutes }} + container: + image: ghcr.io/mesh-llm/mesh-llm-cuda-runner@sha256:8d93de6ba30173e825a16fdecf011f9c632edc6e1259df7289e491b0a05f829d + defaults: + run: + shell: bash env: HF_TOKEN: ${{ secrets.HF_TOKEN }} HUGGING_FACE_HUB_TOKEN: ${{ secrets.HF_TOKEN }} @@ -71,18 +79,8 @@ jobs: ref: ${{ inputs.source_sha || github.sha }} persist-credentials: false - - name: Verify smoke runtime utilities - run: | - set -euo pipefail - for utility in \ - curl \ - jq \ - lsof; do - command -v "$utility" >/dev/null || { - echo "missing smoke runtime utility: $utility" >&2 - exit 1 - } - done + - name: Verify prebuilt smoke environment + run: verify-runner-image public cpu - uses: ./.github/actions/restore-smoke-inputs with: diff --git a/.github/workflows/sdk-smoke.yml b/.github/workflows/sdk-smoke.yml index cc77f8b2e5..b064337f93 100644 --- a/.github/workflows/sdk-smoke.yml +++ b/.github/workflows/sdk-smoke.yml @@ -93,6 +93,17 @@ jobs: timeout-minutes: ${{ inputs.timeout_minutes }} permissions: contents: read + packages: read + # Rust and Kotlin smoke both run on Linux (x86 or arm) and get the + # prebuilt image; Swift stays on bare macos-15, unchanged (that's + # chunk 6). Empty string opts the job out of containerization + # per-run. The truthy value must stay first in the `&&` here -- + # see smoke.yml's container block for what breaks if it isn't. + container: + image: ${{ inputs.sdk_kind != 'swift' && 'ghcr.io/mesh-llm/mesh-llm-cuda-runner@sha256:8d93de6ba30173e825a16fdecf011f9c632edc6e1259df7289e491b0a05f829d' || '' }} + defaults: + run: + shell: bash env: HF_TOKEN: ${{ secrets.HF_TOKEN }} HUGGING_FACE_HUB_TOKEN: ${{ secrets.HF_TOKEN }} @@ -189,21 +200,29 @@ jobs: path: ${{ inputs.kotlin_artifact_path }} - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0 - if: ${{ inputs.sdk_kind == 'rust' }} + # rust is always containerized (only swift opts out), so this stays + # gated off in practice — the image bakes the venv this would shadow. + if: ${{ inputs.sdk_kind == 'rust' && job.container.id == '' }} with: python-version: "3.12" - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + # No JDK is baked into the image (not in verify-runner-image's + # asserted tool list), so this stays ungated. if: ${{ inputs.sdk_kind == 'kotlin' }} with: distribution: temurin java-version: '21' - uses: pnpm/action-setup@b906affcce14559ad1aafd4ab0e942779e9f58b1 # v4.3.0 + # Only the uncontained swift row (macos-15) needs this; rust/kotlin + # get pnpm from the image. + if: job.container.id == '' with: version: 10 - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0 + if: job.container.id == '' with: node-version: "24" cache: pnpm @@ -212,9 +231,9 @@ jobs: - uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 # stable 2026-07-16 if: ${{ inputs.sdk_kind == 'rust' }} - - name: Install Linux SDK dependencies - if: ${{ runner.os == 'Linux' && inputs.sdk_kind == 'rust' }} - run: sudo apt-get update && sudo apt-get install -y build-essential libdbus-1-dev curl jq lsof lld patchelf + - name: Verify prebuilt smoke environment + if: ${{ inputs.sdk_kind != 'swift' }} + run: verify-runner-image public cpu - name: Configure Linux Rust linker if: ${{ runner.os == 'Linux' && inputs.sdk_kind == 'rust' }} @@ -225,26 +244,6 @@ jobs: rustflags = ["-C", "link-arg=-fuse-ld=lld"] EOF - - name: Verify Kotlin smoke runtime utilities - if: ${{ runner.os == 'Linux' && inputs.sdk_kind == 'kotlin' }} - run: | - set -euo pipefail - for utility in \ - bash \ - curl \ - ldd \ - pgrep \ - python3 \ - readelf \ - sort \ - tail \ - xargs; do - command -v "$utility" >/dev/null || { - echo "missing Kotlin smoke runtime utility: $utility" >&2 - exit 1 - } - done - - uses: ./.github/actions/restore-smoke-inputs with: artifact_name: ${{ inputs.artifact_name }} diff --git a/.github/workflows/smoke.yml b/.github/workflows/smoke.yml index 9026faaf28..6840077267 100644 --- a/.github/workflows/smoke.yml +++ b/.github/workflows/smoke.yml @@ -44,6 +44,7 @@ env: permissions: contents: read + packages: read jobs: smoke_tests: @@ -53,9 +54,30 @@ jobs: # uncredentialed PR CUDA smoke. Other credential-bearing smoke stays hosted. runs-on: ${{ inputs.runner == 'gpu-nvidia' && fromJSON('["self-hosted","Linux","X64","amd64","gpu-nvidia","mesh-llm-amd64","mesh-llm"]') || 'ubuntu-24.04' }} timeout-minutes: ${{ inputs.timeout_minutes }} + # Only the GitHub-hosted ubuntu-24.04 path gets prebuilt tooling; the + # gpu-nvidia self-hosted runner keeps running bare-metal, unchanged. + # Empty string opts the job out of containerization per-run. The + # truthy value must come first in the `&&`: `cond && '' || url` always + # evaluates to `url`, because `cond && ''` collapses to the falsy `''` + # and the trailing `||` then overrides it regardless of `cond`. + container: + image: ${{ inputs.runner != 'gpu-nvidia' && 'ghcr.io/mesh-llm/mesh-llm-cuda-runner@sha256:8d93de6ba30173e825a16fdecf011f9c632edc6e1259df7289e491b0a05f829d' || '' }} + defaults: + run: + shell: bash env: HF_TOKEN: ${{ secrets.HF_TOKEN }} HUGGING_FACE_HUB_TOKEN: ${{ secrets.HF_TOKEN }} + # /github/home is a host-side mount owned by the host runner uid; the + # public image's container runs as root (Dockerfile's public stage + # never drops back from `USER root`), and pip refuses a cache dir + # whose owner isn't its own euid, so it otherwise noisily disables + # its cache. This is job-level env, so it applies to every step + # regardless of container -- including the bare-metal gpu-nvidia + # row, where it redirects setup-python's own `cache: pip` off + # `~/.cache/pip` and onto this ephemeral runner's `/tmp` instead. + # Harmless (the runner is single-use), just not container-only. + PIP_CACHE_DIR: /tmp/pip-cache steps: - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 with: @@ -67,6 +89,12 @@ jobs: run: scripts/release-version.sh "${{ inputs.release_tag }}" - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0 + # The prebuilt image bakes a venv on PATH; a host-side setup-python + # both shadows it (silently losing every installed dependency) and, + # in-container, fails outright because its pip cache dir isn't + # owned by the container's (root) euid under the container's HOME. + # Only the uncontained gpu-nvidia row needs this. + if: job.container.id == '' with: python-version: "3.12" cache: pip @@ -75,16 +103,48 @@ jobs: ci/requirements-ci-python.txt - uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0 + if: job.container.id == '' with: node-version: 24 package-manager-cache: false - - name: Install smoke dependencies + - name: Verify prebuilt smoke environment + if: inputs.runner != 'gpu-nvidia' + run: verify-runner-image public cpu + + - name: Install smoke runtime utilities (self-hosted) + if: inputs.runner == 'gpu-nvidia' run: | sudo apt-get update sudo apt-get install -y curl jq lsof - python -m pip install --upgrade pip -r ci/requirements-ci-python.txt - npm install --global openai + + - name: Install smoke dependencies + # The venv baked into the image is pinned to the image's own mesh-llm + # revision; this reconciles it against the repo's current + # requirements even in-container, where it's close to a no-op. + run: python -m pip install --upgrade pip -r ci/requirements-ci-python.txt + + - name: Install OpenAI client + # NOT gated on job.container.id: install-core-tools.sh:83 bakes an + # exact-pinned openai, but that landed in runner-images #20 + # (2026-08-19) and the CPU digest pinned here was built 2026-07-22 + # (verify-runner-image reports mesh_llm_revision 5f341d68). Re-gate + # this only when the CPU digest is promoted past #20 -- and confirm + # it from a green run, not from the image repo's HEAD. + # + # Exact-pinned rather than floating: this step runs with the job + # environment (HF_TOKEN included) and npm lifecycle scripts inherit + # it, so an unreviewed upstream release must not be able to execute + # here. The version tracks the image's own ARG OPENAI_NPM_VERSION + # (mesh-llm-runner-images Dockerfile:25) so that re-gating this step + # once the digest is promoted is a no-op rather than a swap. + env: + # npm lifecycle scripts inherit the job environment, and neither + # token is needed to install a client library. Deny both to this + # step rather than relying on the version pin alone. + HF_TOKEN: "" + HUGGING_FACE_HUB_TOKEN: "" + run: npm install --global openai@7.5.0 - name: Install CUDA smoke runtime libraries if: inputs.runner == 'gpu-nvidia' @@ -158,6 +218,10 @@ jobs: run: scripts/release-version.sh "${{ inputs.release_tag }}" - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0 + # This job never containerizes, so this is always true today; gated + # for the same reason as the linux job's — a single expression that + # can't drift from whatever decides containerization. + if: job.container.id == '' with: python-version: "3.12" cache: pip @@ -166,14 +230,27 @@ jobs: ci/requirements-ci-python.txt - uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0 + if: job.container.id == '' with: node-version: 24 package-manager-cache: false - name: Install smoke dependencies - run: | - python -m pip install --upgrade pip -r ci/requirements-ci-python.txt - npm install --global openai + run: python -m pip install --upgrade pip -r ci/requirements-ci-python.txt + + - name: Install OpenAI client (uncontainerized rows only) + # This job never containerizes (see the setup-python/setup-node gates + # above); gated the same way for consistency, not because it varies. + # Pinned to the same version as the Linux job above -- see the + # rationale there. + if: job.container.id == '' + env: + # npm lifecycle scripts inherit the job environment, and neither + # token is needed to install a client library. Deny both to this + # step rather than relying on the version pin alone. + HF_TOKEN: "" + HUGGING_FACE_HUB_TOKEN: "" + run: npm install --global openai@7.5.0 - uses: ./.github/actions/restore-smoke-inputs with: diff --git a/.github/workflows/static-abi-artifact.yml b/.github/workflows/static-abi-artifact.yml index 1e32ff7c5a..4f3743c306 100644 --- a/.github/workflows/static-abi-artifact.yml +++ b/.github/workflows/static-abi-artifact.yml @@ -161,7 +161,7 @@ jobs: persist-credentials: false - name: Verify prebuilt static ABI environment - run: verify-runner-image public + run: verify-runner-image public cpu - name: Resolve static ABI toolchain epoch id: native_toolchain diff --git a/.github/workflows/swift-sdk-artifact.yml b/.github/workflows/swift-sdk-artifact.yml index 8d68bc567f..e1c2fc6b7d 100644 --- a/.github/workflows/swift-sdk-artifact.yml +++ b/.github/workflows/swift-sdk-artifact.yml @@ -169,7 +169,9 @@ jobs: allow_native_github_cache: ${{ needs.runner_policy.outputs.allow_native_github_cache }} - name: Install macOS SDK build dependencies - run: brew install cmake ninja lld + run: brew install cmake ninja + - name: Install macOS SDK linker + uses: ./.github/actions/setup-macos-lld - name: Resolve macOS native toolchain epoch id: native_toolchain diff --git a/.github/workflows/website-pages.yml b/.github/workflows/website-pages.yml index bc9cdbbabf..ef4efea6da 100644 --- a/.github/workflows/website-pages.yml +++ b/.github/workflows/website-pages.yml @@ -19,6 +19,7 @@ on: permissions: contents: read + packages: read concurrency: group: public-website-pages @@ -30,24 +31,26 @@ jobs: if: github.event_name != 'workflow_dispatch' || github.ref == 'refs/heads/main' runs-on: ubuntu-24.04 timeout-minutes: 40 + container: + image: ghcr.io/mesh-llm/mesh-llm-cuda-runner@sha256:1c73f0f2e4b2ce657cc9dc49c9c27374f3c364af19f884f1970331c3689da5d2 steps: - name: Checkout repository uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 - name: Set up Rust + # Kept deliberately: the image's rustc comes from `rustup ... + # --default-toolchain stable` at image-build time, an unpinned + # moving target, not this repo's pinned toolchain SHA. `just + # crate-docs` (below) runs `cargo doc` for the published crate API + # docs this job deploys to Pages, and this workflow has no + # pre-merge harness coverage at all (push-to-main, path-filtered) -- + # so a silent toolchain drift here would ship unnoticed. Re-pins on + # top of the baked toolchain; CARGO_HOME/RUSTUP_HOME are baked ENV + # so this resolves the same install location regardless of the + # container's HOME. uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 # stable 2026-07-16 - - name: Install just - uses: taiki-e/install-action@3d23c1bbdafe696dfccad2664945a04f47d03dc3 # just - - - name: Set up Node.js - uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0 - with: - node-version: 24 - cache: npm - cache-dependency-path: website/package-lock.json - - name: Install website dependencies working-directory: website run: npm ci @@ -64,6 +67,9 @@ jobs: run: just crate-docs - name: Stage Pages artifact + # shell: bash is required: this job's container: resolves the + # default run: shell to sh -e {0}, and dash has no -o pipefail. + shell: bash run: | set -euo pipefail diff --git a/ci/ci.md b/ci/ci.md index ad0f97e015..aacde8b531 100644 --- a/ci/ci.md +++ b/ci/ci.md @@ -472,8 +472,8 @@ The implemented policy uses that isolation selectively: | Cargo `target` directories | Restore trusted main, never save from PR | A rerun reuses the latest compatible main cache, but not objects compiled by the earlier PR run | | Static Linux ABI and Swift native ABI | Exact PR-scoped cache on miss | Same-PR reruns reuse the verified native input when its full recipe/toolchain key is unchanged | | macOS Metal unit ABI and Windows native ABI | Exact PR-scoped cache on miss | Same-PR reruns avoid the native rebuild; no restore prefixes cross an ABI boundary | -| Console pnpm store | Website is the sole publisher; platform UI jobs restore only | Avoids four platform workflows racing to upload the same entry; later same-PR runs reuse a lockfile-keyed store | -| Website npm store | Website-only lockfile-keyed cache | Later same-PR website runs avoid downloading the unchanged dependency store | +| Console pnpm store | `ui_quality` is the sole publisher; `ui_e2e` restores only | Avoids the two `ci-web-slice.yml` UI jobs racing to upload the same entry; later same-PR runs reuse a lockfile-keyed store | +| Website npm store | None -- the `website` job runs in the prebuilt `public web` image (baked npm/node) with no bare-metal row, so its `setup-node` cache was deleted outright rather than kept | Every run does a fresh `npm ci`; no cache to invalidate or race | | GitHub artifacts | Never used as cross-run caches | Immutable producers/consumers remain correct within one run; reruns recreate run-scoped artifacts | Outside the bounded exception, a Depot-selected run emits diff --git a/scripts/tests/test_ci_artifact_actions.py b/scripts/tests/test_ci_artifact_actions.py index eb5edb847b..d09a9d7fe7 100644 --- a/scripts/tests/test_ci_artifact_actions.py +++ b/scripts/tests/test_ci_artifact_actions.py @@ -2651,9 +2651,6 @@ def step_block(workflow: str, marker: str) -> str: r"^ [A-Za-z0-9_]+:\n(?: [^\n]*\n){0,4} if:.*allow_native_github_cache", ) - website = ( - ROOT / ".github" / "workflows" / "ci-web-slice.yml" - ).read_text(encoding="utf-8") swift = ( ROOT / ".github" / "workflows" / "swift-sdk-artifact.yml" ).read_text(encoding="utf-8") @@ -2669,14 +2666,12 @@ def step_block(workflow: str, marker: str) -> str: native_cache_expression = ( "needs.runner_policy.outputs.allow_native_github_cache == 'true'" ) - self.assertIn( - f"cache: ${{{{ {native_cache_expression} && 'npm' || '' }}}}", - website, - ) - self.assertIn( - f"package-manager-cache: ${{{{ {native_cache_expression} }}}}", - website, - ) + # ci-web-slice.yml's `website` job runs in the prebuilt public-web + # image (no bare-metal row), so its setup-node native-cache + # consumer was deleted outright rather than gated -- there is + # nothing left in that job for the depot/native cache policy to + # govern. See `eligible_consumers["ci-web-slice.yml"]` above for + # the jobs in this file that still participate (ui_quality, ui_e2e). self.assertIn( f"cache: ${{{{ {native_cache_expression} && 'pnpm' || '' }}}}", swift, diff --git a/scripts/tests/test_ci_workflow_artifacts.py b/scripts/tests/test_ci_workflow_artifacts.py index 0a0f477d5b..1e7e71f70a 100644 --- a/scripts/tests/test_ci_workflow_artifacts.py +++ b/scripts/tests/test_ci_workflow_artifacts.py @@ -178,14 +178,13 @@ def test_ui_cache_and_website_dependencies_are_explicit(self): self.assertIn("uses: actions/cache/restore@", ui) self.assertNotIn("uses: actions/cache/save@", ui) self.assertIn("uses: actions/cache/save@", web) - self.assertIn( - "cache: ${{ needs.runner_policy.outputs.allow_native_github_cache == 'true' && 'npm' || '' }}", - web, - ) - self.assertIn("website/package-lock.json", web) + # The `website` job runs in the prebuilt public-web image with no + # bare-metal row, so setup-node's own npm cache and the `just` + # install-action were deleted outright (both are baked in the + # image) rather than gated -- unlike ui_quality/ui_e2e above, this + # job has no native-cache consumer left to assert on. self.assertIn("working-directory: website", web) self.assertIn("run: npm ci", web) - self.assertIn("uses: taiki-e/install-action@", web) if __name__ == "__main__": diff --git a/scripts/tests/test_ci_workflow_container_shell_contract.py b/scripts/tests/test_ci_workflow_container_shell_contract.py new file mode 100644 index 0000000000..622a65b895 --- /dev/null +++ b/scripts/tests/test_ci_workflow_container_shell_contract.py @@ -0,0 +1,101 @@ +from __future__ import annotations + +from pathlib import Path +import re +import unittest + +import yaml + + +ROOT = Path(__file__).resolve().parents[2] +WORKFLOWS_DIR = ROOT / ".github" / "workflows" + +# Bash-only constructs that fail outright under dash/sh, keyed by a short +# human label for the failure message. This is a best-effort sweep, not an +# exhaustive shell grammar -- it exists to catch the same class of mistake +# that already bit this repo twice, not to replace shellcheck. +_BASHISM_PATTERNS: list[tuple[re.Pattern[str], str]] = [ + (re.compile(r"<<<"), "here-string (<<<)"), + (re.compile(r"\bpipefail\b"), "set -o pipefail"), + (re.compile(r"\[\["), "[[ ... ]] test"), + # Deliberately NOT listed: $(( )) arithmetic expansion. It is POSIX + # (Shell Command Language 2.6.4) and dash evaluates it correctly, so + # flagging it would reject valid sh steps and force a spurious + # `shell: bash`. + (re.compile(r"\$\{\w+(//|\^\^|,,)"), "parameter expansion (${v//}/${v^^}/${v,,})"), + (re.compile(r"&>"), "&> redirection"), + (re.compile(r"\$RANDOM\b"), "$RANDOM"), + (re.compile(r"(?:^|[;\s])source\s+\S"), "source builtin"), + (re.compile(r"^\s*function\s+\w+", re.MULTILINE), "function keyword"), + (re.compile(r"\w+\+=\("), "array += assignment"), + (re.compile(r"^\s*declare\s+-a\b", re.MULTILINE), "declare -a array"), + (re.compile(r"\b(?:mapfile|readarray)\b"), "mapfile/readarray"), + (re.compile(r"\becho\s+-e\b"), "echo -e"), +] + + +def _load_workflow(path: Path) -> dict: + return yaml.safe_load(path.read_text(encoding="utf-8")) or {} + + +def _default_shell(defaults) -> str | None: + if not isinstance(defaults, dict): + return None + run = defaults.get("run") + if not isinstance(run, dict): + return None + shell = run.get("shell") + return shell.lower() if isinstance(shell, str) else None + + +class CiWorkflowContainerShellContractTests(unittest.TestCase): + """GitHub Actions resolves the default `run:` shell inside a `container:` + job to `sh -e {0}`, not `bash -e {0}` -- bare-metal Linux/macOS runners + default to bash, so this only changes once a job gains a `container:` + block. A step using a bashism there fails at runtime with a dash/sh + syntax error that `actionlint` cannot catch (its shellcheck integration + assumes bash). This walks every job with a `container:` block and flags + any `run:` step containing a bashism that has not declared `shell: bash` + for itself, its job, or its workflow. See ci-web-slice.yml's `ui_e2e` + preflight (<<<) and website-pages.yml's `Stage Pages artifact` + (set -euo pipefail) -- both hit this in the same PR.""" + + def test_bashisms_in_container_jobs_declare_shell_bash(self) -> None: + violations = [] + for path in sorted(WORKFLOWS_DIR.glob("*.yml")): + doc = _load_workflow(path) + workflow_shell = _default_shell(doc.get("defaults")) + jobs = doc.get("jobs") or {} + for job_name, job in jobs.items(): + if not isinstance(job, dict) or not job.get("container"): + continue + job_shell = _default_shell(job.get("defaults")) or workflow_shell + for step in job.get("steps") or []: + if not isinstance(step, dict) or "run" not in step: + continue + step_shell = step.get("shell") + step_shell = step_shell.lower() if isinstance(step_shell, str) else job_shell + if step_shell == "bash": + continue + run_text = step.get("run") or "" + step_label = step.get("name", "") + for pattern, label in _BASHISM_PATTERNS: + if pattern.search(run_text): + violations.append( + f"{path.name} :: {job_name} :: {step_label!r} uses " + f"{label} but the effective shell in this container " + "job is sh, not bash -- add `shell: bash` to the step." + ) + break + + self.assertEqual( + [], + violations, + "Containerized jobs resolve the default `run:` shell to sh, not " + "bash. These steps use a bashism without declaring `shell: bash`:\n" + + "\n".join(violations), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/tests/test_ci_workflow_permission_contract.py b/scripts/tests/test_ci_workflow_permission_contract.py new file mode 100644 index 0000000000..75e229ed56 --- /dev/null +++ b/scripts/tests/test_ci_workflow_permission_contract.py @@ -0,0 +1,264 @@ +from __future__ import annotations + +from pathlib import Path +import re +import unittest + +import yaml + + +ROOT = Path(__file__).resolve().parents[2] +WORKFLOWS_DIR = ROOT / ".github" / "workflows" + +# GitHub Actions permission levels, ranked. A caller satisfies a callee only if +# it grants at least the level the callee asks for -- `contents: read` does NOT +# satisfy `contents: write`, and GitHub rejects that downgrade at run creation +# exactly like a missing scope. Comparing scope *names* alone accepts it +# silently, so the level has to survive into the comparison. +_LEVEL_RANK = {"none": 0, "read": 1, "write": 2} +_GRANTED_LEVELS = ("read", "write") + +_LOCAL_USES_RE = re.compile(r"^\./\.github/workflows/([^@\s]+\.ya?ml)$") + + +def _load_workflow(path: Path) -> dict: + return yaml.safe_load(path.read_text(encoding="utf-8")) or {} + + +def _rank(level: str | None) -> int: + return _LEVEL_RANK.get(level or "none", 0) + + +class _AllScopes: + """`permissions: read-all` / `write-all` -- one blanket level across every + scope. Modelled rather than skipped, because as a *grant* it is perfectly + enumerable: `write-all` satisfies any request, and `read-all` satisfies a + `read` request but not a `write` one. (As a *request* it is still opaque -- + see _requested_scopes -- since it names no scopes to check against.)""" + + def __init__(self, level: str) -> None: + self.level = level + + def get(self, scope: str, default: str = "none") -> str: + return self.level + + def __repr__(self) -> str: + return f"{{every scope: {self.level}}}" + + +def _format(scopes) -> str: + if isinstance(scopes, _AllScopes): + return repr(scopes) + return "{" + ", ".join(f"{scope}: {level}" for scope, level in sorted(scopes.items())) + "}" + + +def _scope_levels(permissions) -> dict[str, str] | _AllScopes | None: + """Map each scope in a permissions: value to the level it is set to, or + None if no explicit block is present (meaning the check does not apply -- + GitHub falls back to the repo/org default token, which this test cannot + see). `read-all`/`write-all` become an _AllScopes blanket level. Scopes set + to `none` are dropped: they neither grant nor request.""" + if permissions is None: + return None + if isinstance(permissions, str): + if permissions == "read-all": + return _AllScopes("read") + if permissions == "write-all": + return _AllScopes("write") + return {} + return {scope: level for scope, level in permissions.items() if level in _GRANTED_LEVELS} + + +def _requested_scopes(doc: dict) -> dict[str, str] | None: + """Every scope a reusable workflow can request, at the highest level it + asks for anywhere: the workflow-level permissions block merged with every + explicit job-level block. GitHub evaluates permissions at both levels, so a + callee that declares `packages: read` on one job alone still needs its + caller to grant it -- reading only the workflow-level block returns None + there and silently skips the caller edge, which is exactly the hop this + test exists to cover. Where a scope appears in more than one block the + highest level wins, because the caller has to satisfy the strictest + request. Returns None when nothing explicit is declared anywhere, or when + any level uses read-all/write-all (unenumerable -- do not assert).""" + blocks = [doc.get("permissions")] + jobs = doc.get("jobs") or {} + if isinstance(jobs, dict): + for job in jobs.values(): + if isinstance(job, dict): + blocks.append(job.get("permissions")) + + requested: dict[str, str] | None = None + for block in blocks: + if block is None: + continue + levels = _scope_levels(block) + if levels is None or isinstance(levels, _AllScopes): + # Unenumerable as a request: it names no scopes to hold the caller + # to, so asserting anything here would be invention. + return None + if requested is None: + requested = {} + for scope, level in levels.items(): + if _rank(level) > _rank(requested.get(scope)): + requested[scope] = level + return requested + + +def _unsatisfied(requested: dict[str, str], granted) -> list[str]: + """Scopes the caller fails to cover, either because it omits them or + because it grants a weaker level than the callee requests.""" + return [ + f"{scope}: needs {level}, has {granted.get(scope, 'none')}" + for scope, level in sorted(requested.items()) + if _rank(level) > _rank(granted.get(scope)) + ] + + +class CiWorkflowPermissionContractTests(unittest.TestCase): + """A called reusable workflow can only use permissions its caller job + actually grants it — GitHub rejects the run at creation time otherwise + (a zero-job startup_failure, invisible to actionlint and to PR CI since + the caller's own PR run never requests the scope the callee needs until + that specific callee executes). This walks every local + `uses: ./.github/workflows/X.yml` edge in the repo and asserts the + caller's effective permissions (job-level, else workflow-level) cover + every scope X.yml requests, at no less than the level it requests. + """ + + @classmethod + def setUpClass(cls) -> None: + cls.workflows: dict[str, dict] = { + path.name: _load_workflow(path) for path in sorted( + [*WORKFLOWS_DIR.glob("*.yml"), *WORKFLOWS_DIR.glob("*.yaml")] + ) + } + cls.requested: dict[str, dict[str, str] | None] = { + name: _requested_scopes(doc) + for name, doc in cls.workflows.items() + if isinstance(doc.get(True, doc.get("on")), dict) + and "workflow_call" in doc.get(True, doc.get("on")) + } + + def test_every_local_reusable_call_site_grants_the_callees_permissions(self) -> None: + violations = [] + for caller_name, doc in self.workflows.items(): + jobs = doc.get("jobs") or {} + for job_name, job in jobs.items(): + uses = job.get("uses") if isinstance(job, dict) else None + if not isinstance(uses, str): + continue + match = _LOCAL_USES_RE.match(uses) + if not match: + continue + callee_name = match.group(1) + requested = self.requested.get(callee_name) + if not requested: + continue # callee declares no permissions, or isn't a workflow_call target we tracked + + job_permissions = job.get("permissions") + if job_permissions is not None: + granted = _scope_levels(job_permissions) + else: + granted = _scope_levels(doc.get("permissions")) + + if granted is None: + continue # no explicit block at the effective level — can't assert, GitHub uses the default token + + unsatisfied = _unsatisfied(requested, granted) + if unsatisfied: + violations.append( + f"{caller_name}:{job_name} -> {callee_name} requests " + f"{_format(requested)} but grants {_format(granted)} " + f"({'; '.join(unsatisfied)})" + ) + + self.assertEqual( + [], + violations, + "Reusable workflow call sites must grant every permission scope their " + "callee requests, at every hop and at no less than the requested level — " + "GitHub does not let a called workflow reach past what its immediate " + "caller job declares:\n" + "\n".join(violations), + ) + + def test_a_read_grant_does_not_satisfy_a_write_request(self) -> None: + """Regression: the contract compares levels, not just scope names. A + caller granting `contents: read` to a callee that needs + `contents: write` is the same run-creation failure as omitting the + scope, so it must be reported, not passed over.""" + requested = _requested_scopes({"permissions": {"contents": "write"}}) + self.assertEqual({"contents": "write"}, requested) + + self.assertEqual( + ["contents: needs write, has read"], + _unsatisfied(requested, {"contents": "read"}), + "a read grant must not satisfy a write request", + ) + self.assertEqual( + [], + _unsatisfied(requested, {"contents": "write"}), + "an equal grant must satisfy the request", + ) + self.assertEqual( + [], + _unsatisfied(_requested_scopes({"permissions": {"contents": "read"}}), {"contents": "write"}), + "a stronger grant must satisfy a weaker request", + ) + self.assertEqual( + ["contents: needs read, has none"], + _unsatisfied(_requested_scopes({"permissions": {"contents": "read"}}), {"packages": "read"}), + "an omitted scope must still be reported", + ) + + def test_all_scope_caller_grants_are_modelled_not_skipped(self) -> None: + """A caller declaring `read-all`/`write-all` used to return None and + skip the edge entirely. `write-all` genuinely satisfies anything; + `read-all` genuinely does not satisfy a `write` request, and skipping + it hid exactly the run-creation failure this test exists to catch.""" + needs_write = _requested_scopes({"permissions": {"contents": "write"}}) + needs_read = _requested_scopes({"permissions": {"contents": "read"}}) + + write_all = _scope_levels("write-all") + read_all = _scope_levels("read-all") + self.assertIsInstance(write_all, _AllScopes) + self.assertIsInstance(read_all, _AllScopes) + + self.assertEqual([], _unsatisfied(needs_write, write_all), "write-all satisfies write") + self.assertEqual([], _unsatisfied(needs_read, write_all), "write-all satisfies read") + self.assertEqual([], _unsatisfied(needs_read, read_all), "read-all satisfies read") + self.assertEqual( + ["contents: needs write, has read"], + _unsatisfied(needs_write, read_all), + "read-all must NOT satisfy a write request", + ) + + def test_an_all_scope_callee_request_stays_unenumerable(self) -> None: + """The same forms are opaque in the other direction: a callee asking + `write-all` names no scopes, so there is nothing to hold the caller to + and the contract declines to assert rather than invent a scope list.""" + self.assertIsNone(_requested_scopes({"permissions": "write-all"})) + self.assertIsNone(_requested_scopes({"permissions": "read-all"})) + self.assertIsNone( + _requested_scopes({ + "permissions": {"contents": "read"}, + "jobs": {"a": {"permissions": "write-all"}}, + }), + "one all-scope job block makes the whole request unenumerable", + ) + + def test_the_strictest_level_wins_when_a_scope_is_declared_twice(self) -> None: + """A callee asking `contents: read` at the workflow level and + `contents: write` on one job needs write from its caller. Merging by + scope name alone would keep whichever block was seen last.""" + doc = { + "permissions": {"contents": "read"}, + "jobs": { + "a": {"permissions": {"contents": "write"}}, + "b": {"permissions": {"packages": "read", "contents": "none"}}, + }, + } + self.assertEqual({"contents": "write", "packages": "read"}, _requested_scopes(doc)) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/tests/test_ci_workflow_ternary_contract.py b/scripts/tests/test_ci_workflow_ternary_contract.py new file mode 100644 index 0000000000..da827bb228 --- /dev/null +++ b/scripts/tests/test_ci_workflow_ternary_contract.py @@ -0,0 +1,56 @@ +from __future__ import annotations + +from pathlib import Path +import re +import unittest + + +ROOT = Path(__file__).resolve().parents[2] +WORKFLOWS_DIR = ROOT / ".github" / "workflows" + +_EXPRESSION_RE = re.compile(r"\$\{\{(.*?)\}\}", re.DOTALL) + +# `cond && a || b` is GitHub Actions' idiom for a ternary, built out of +# JS-style short-circuit `&&`/`||`: `&&` returns its second operand only +# when the first is truthy, `||` returns its first operand only when it's +# truthy. That means the `&&` branch (`a`) must never itself be falsy -- +# if it is, `cond && a` collapses to the falsy `a`, and the trailing `||` +# then overrides it with `b` unconditionally, regardless of `cond`. The +# three literals that can trigger this are '', "", 0, and false. +_FALSY_AND_BRANCH_RE = re.compile(r"&&\s*(''|\"\"|0|false)\s*\|\|") + + +class CiWorkflowTernaryContractTests(unittest.TestCase): + """Catches the `cond && || fallback` pitfall in workflow + expressions: a ternary written with `&&`/`||` where the "true" branch is + itself a falsy literal always evaluates to the fallback, never the + intended value. See smoke.yml's `container: image:` line for the case + this caught in production -- the `gpu-nvidia` branch of that ternary + (`inputs.runner == 'gpu-nvidia' && '' || url`) could never actually + produce an empty image.""" + + def test_no_ternary_has_a_falsy_literal_in_the_and_branch(self) -> None: + violations = [] + for path in sorted(WORKFLOWS_DIR.glob("*.yml")): + text = path.read_text(encoding="utf-8") + for match in _EXPRESSION_RE.finditer(text): + expression = match.group(1) + if _FALSY_AND_BRANCH_RE.search(expression): + line = text.count("\n", 0, match.start()) + 1 + violations.append( + f"{path.name}:{line}: {expression.strip()!r} -- the `&&` " + "branch is a falsy literal, so `||` always overrides it; " + "put the non-empty/truthy value first instead." + ) + + self.assertEqual( + [], + violations, + "Workflow ternaries must not put a falsy literal ('', \"\", 0, false) " + "in the `&&` branch -- it makes the `||` fallback unconditional:\n" + + "\n".join(violations), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/tests/test_pr_workflow_artifacts.py b/scripts/tests/test_pr_workflow_artifacts.py index 30694135fa..63b61afab2 100644 --- a/scripts/tests/test_pr_workflow_artifacts.py +++ b/scripts/tests/test_pr_workflow_artifacts.py @@ -252,11 +252,10 @@ def test_pr_cache_publishers_are_exact_and_bounded(self): website = self.workflow("ci-web-slice.yml") self.assertNotIn("name: Save pnpm store", ui_artifact) self.assertEqual(1, website.count("name: Save pnpm store")) - self.assertIn( - "cache: ${{ needs.runner_policy.outputs.allow_native_github_cache == 'true' && 'npm' || '' }}", - website, - ) - self.assertIn("website/package-lock.json", website) + # The `website` job itself runs in the prebuilt public-web image with + # no bare-metal row, so it has no native-cache-gated npm consumer + # left to publish or bound -- setup-node's own cache was deleted + # outright, not gated. windows = self.workflow("ci-windows-runtime-slice.yml") self.assertIn("name: Save exact PR-scoped Windows ABI build", windows)