diff --git a/.github/workflows/ecosystem.yml b/.github/workflows/ecosystem.yml new file mode 100644 index 000000000..3286be683 --- /dev/null +++ b/.github/workflows/ecosystem.yml @@ -0,0 +1,42 @@ +name: Ecosystem + +# Runs the built CLI against real third-party SvelteKit apps and asserts only that it did not fall +# over — never a score or a count. Design: docs/superpowers/specs/2026-08-16-ecosystem-smoke-design.md +# +# Never PR-blocking: the corpus tracks upstream default branches, so a repo moving or a clone +# failing is not a reason to hold a merge. The pull_request trigger is scoped to this job's own +# files so a corpus edit is validated by the thing it edits. +on: + schedule: + - cron: '17 4 * * 1' + workflow_dispatch: + pull_request: + paths: + - 'scripts/ecosystem-smoke.mjs' + - '.github/workflows/ecosystem.yml' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +# The job clones untrusted third-party code. Nothing it can reach is worth anything. +permissions: + contents: read + +jobs: + ecosystem: + runs-on: ubuntu-latest + # A real run is under two minutes. This is a backstop sized above the script's own worst + # case — eight targets each hitting both per-target limits — so a bad week still ends with the + # script's per-target report rather than a mid-run kill. + timeout-minutes: 40 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Setup Node.js and dependencies + uses: ./.github/workflows/setup-node + - name: Build packages + run: pnpm build + - name: Run the ecosystem smoke + run: node scripts/ecosystem-smoke.mjs diff --git a/AGENTS.md b/AGENTS.md index 9e0acfed6..1dbd35dde 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,11 +14,12 @@ svelte-vitals is a static code-health checker for SvelteKit — not a runtime We | Typecheck | `pnpm typecheck` | `pnpm -r typecheck` | | Test | `pnpm test` | `pnpm build && pnpm -r test` (vitest) — builds first because packages/cli's tests import @svelte-vitals/core from its built dist | | Floor smoke | `pnpm smoke` | needs `pnpm build` first — it runs the built `dist` under a bare `node`; locally that is the devEngines Node, not the floor, so the floor claim is what CI's `floor-smoke` job (pinned to 22.13.0) adds | +| Ecosystem | `pnpm ecosystem` | needs `pnpm build` first — clones the third-party SvelteKit apps listed in `scripts/ecosystem-smoke.mjs` and asserts only "no crash, exit ∈ {0,1}, report parses"; scheduled weekly, never PR-blocking | | Lint | `pnpm lint` | `oxlint .` + `oxfmt --check .` | | Format | `pnpm format` | `oxfmt --write .` | | Publish checks | `pnpm check:publish` | publint + attw (`--profile esm-only`) | -CI (`.github/workflows/ci.yml`) runs five jobs: `lint`, `check` (build + typecheck + check:publish), `test`, `floor-smoke`, `docs`. Run the relevant verify commands yourself and confirm they pass **before** claiming a task is complete. +CI (`.github/workflows/ci.yml`) runs five jobs: `lint`, `check` (build + typecheck + check:publish), `test`, `floor-smoke`, `docs`. A separate `.github/workflows/ecosystem.yml` runs the ecosystem smoke weekly. Run the relevant verify commands yourself and confirm they pass **before** claiming a task is complete. ## Package map diff --git a/docs/superpowers/specs/2026-08-16-ecosystem-smoke-design.md b/docs/superpowers/specs/2026-08-16-ecosystem-smoke-design.md new file mode 100644 index 000000000..57168c50a --- /dev/null +++ b/docs/superpowers/specs/2026-08-16-ecosystem-smoke-design.md @@ -0,0 +1,140 @@ +# Ecosystem smoke — design + +Phase B-3 of `2026-08-16-v1-roadmap.md`. A scheduled job that runs the built CLI against real +third-party SvelteKit apps and asserts only that it did not fall over. The roadmap already fixed the +assertions ("no crash, exit ∈ {0,1}, report parses — never counts"); this records the decisions it +left open. + +## Why, and — precisely — what it does not cover + +The recent run of engine bugs all came from pointing the tool at code nobody wrote for it. This job +turns one class of that luck into a standing net, and it is worth being exact about which class, +because the ratio is not flattering: + +| bug | class | caught here? | +| ------------------------------------------- | -------------- | -------------- | +| `#508` `