diff --git a/.coderabbit.yml b/.coderabbit.yml index a21f159ed..993639046 100644 --- a/.coderabbit.yml +++ b/.coderabbit.yml @@ -4,10 +4,12 @@ # Review instructions are the conventions themselves — the ones a test cannot check. language: 'en-US' +# Hard limit: 250 characters. Over it the whole file fails to parse and CodeRabbit silently +# reviews with its defaults, so every instruction below is dropped too. tone_instructions: >- - Terse and concrete. Lead with the rule, cite the file that states it, skip nitpicks Biome already - catches. The primary user of this codebase is an AI agent: block anything that makes a failure - unreadable to one — a bare Error, a missing fix command, a command without --json. + Terse and concrete. Lead with the rule, cite the file that states it, skip nitpicks Biome + catches. The primary user here is an AI agent: block whatever makes a failure unreadable to + one — a bare Error, a missing fix, a command without --json. early_access: true enable_free_tier: true diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 60262c993..26063ea45 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -27,6 +27,8 @@ jobs: timeout-minutes: 5 steps: - uses: actions/checkout@v7 + with: + persist-credentials: false - uses: oven-sh/setup-bun@v2 with: bun-version: latest @@ -44,6 +46,8 @@ jobs: timeout-minutes: 8 steps: - uses: actions/checkout@v7 + with: + persist-credentials: false - uses: oven-sh/setup-bun@v2 with: bun-version: latest @@ -61,6 +65,8 @@ jobs: timeout-minutes: 5 steps: - uses: actions/checkout@v7 + with: + persist-credentials: false - uses: oven-sh/setup-bun@v2 with: bun-version: latest @@ -73,6 +79,8 @@ jobs: timeout-minutes: 10 steps: - uses: actions/checkout@v7 + with: + persist-credentials: false - uses: oven-sh/setup-bun@v2 with: bun-version: latest @@ -83,12 +91,9 @@ jobs: restore-keys: bun-${{ runner.os }}- - run: bun install --frozen-lockfile - name: bun test - # `bun run test`, not bare `bun test`: CI must run the repo's own gate, not a - # different command. The script's ignore patterns drop the `e2e/` suites, which - # bind a real socket and are opt-in behind ULTIMATE_TEST_ALLOW_NET=1, and the - # dummy app's contract/live/job/eval suites, which request fixtures nothing in - # the repo registers (#9). Bare `bun test` picked both up, which is why `main` - # has been red since 2026-07-30 and every PR inherited it. + # No opt-in exclusions: the framework's e2e/contract/live/job/eval suites all run, every + # time. The reference app's suites are not here — they run in `reference-app-verify`, + # against the app's own preload and fixtures. run: bun run test # The whole gate, exactly as `x verify` runs it. Green here means shippable. @@ -98,6 +103,8 @@ jobs: needs: [lint, typecheck, boundaries, test] steps: - uses: actions/checkout@v7 + with: + persist-credentials: false - uses: oven-sh/setup-bun@v2 with: bun-version: latest @@ -109,3 +116,43 @@ jobs: - run: bun install --frozen-lockfile - name: x verify run: bun run scripts/verify.ts --json + + # The reference app runs the same 15-step gate a generated app would. Not in `verify`'s + # `needs`: the framework gate and the app gate are two different contracts, checked + # independently — a red app doesn't hide a green framework or vice versa. That is also why + # the app is absent from the `typecheck` and `test` jobs above: its typecheck and its suites + # are steps 1 and 6-11 of the gate below, so running them twice only decides which job's + # redness you notice first. + # + # Advisory, not blocking. As of 2026-08 Postly is written against the finished API — the fluent + # `action` surface, `defineMail`/`defineService`, the `page`/`subscribe`/`evaluate` fixtures — + # and the framework has not shipped it yet, so every failure here is drift the roadmap already + # owns rather than a regression this PR introduced. Blocking on it would make red the resting + # state, which is how a gate stops being read. `continue-on-error` comes off at milestone 9 in + # docs/idea/14-roadmap.md — the last of those surfaces to land — when app and framework agree. + reference-app-verify: + runs-on: ubuntu-latest + timeout-minutes: 12 + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - uses: oven-sh/setup-bun@v2 + with: + bun-version: latest + - uses: actions/cache@v6 + with: + path: ~/.bun/install/cache + key: bun-${{ runner.os }}-${{ hashFiles('bun.lock') }} + restore-keys: bun-${{ runner.os }}- + - run: bun install --frozen-lockfile + - name: x verify (examples/dummy) + # `continue-on-error` on the STEP, not the job: on the job it leaves the check itself red + # for as long as the drift above lasts, and a check that is always red is a check nobody + # reads — the exact failure mode this PR exists to remove. On the step the run is marked + # with a warning, and the whole 15-step table, every failing step and every fix line stay + # in the log. Human render, not `--json`: the reader of an advisory result is a person or + # an agent scanning for which step went red, not a parser. + continue-on-error: true + working-directory: examples/dummy + run: bun run ../../packages/cli/src/bin.ts verify diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index 42bd285c4..34d200bc2 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -24,6 +24,8 @@ jobs: timeout-minutes: 10 steps: - uses: actions/checkout@v7 + with: + persist-credentials: false - uses: oven-sh/setup-bun@v2 with: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a1fbfa475..cb6fcef3a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -33,6 +33,8 @@ jobs: timeout-minutes: 20 steps: - uses: actions/checkout@v7 + with: + persist-credentials: false # No `registry-url:` on purpose. It writes an .npmrc containing # //registry.npmjs.org/:_authToken=${NODE_AUTH_TOKEN} diff --git a/.github/workflows/wiki.yml b/.github/workflows/wiki.yml index b270d97e6..8cdb6bb13 100644 --- a/.github/workflows/wiki.yml +++ b/.github/workflows/wiki.yml @@ -22,6 +22,8 @@ jobs: timeout-minutes: 5 steps: - uses: actions/checkout@v7 + with: + persist-credentials: false - name: detect wiki sources id: wiki diff --git a/CLAUDE.md b/CLAUDE.md index 88968f7c4..7f8a0bf17 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,10 +29,10 @@ CLI binary: `x`. npm scope: `@ultimat3`. Import paths: `@ultimat3/`. | Task | Command | |---|---| | install | `bun install` | -| **the gate** | `bun run verify` — typecheck + lint + boundaries + tests. Green = shippable. | +| **the gate** | `bun run verify` — `x verify` at the repo root: typecheck, lint, boundaries, sizes, shape, every test type, drift, contracts, budgets, manifest. Green = shippable. | | typecheck | `bun run typecheck` | | lint | `bun run lint` · fix: `bun run lint:fix` | -| test (all) | `bun run test` — bare `bun test` also collects the opt-in `e2e/` and dummy-app fixture suites (#9) | +| test (all) | `bun run test` — every framework suite, opt-in ones included. The reference app is gated separately: `cd examples/dummy && bun run ../../packages/cli/src/bin.ts verify` | | test (one file) | `bun test packages/core/src/errors.test.ts` | | test (one name) | `bun test -t 'formats the fix line'` | | import boundaries | `bun run boundaries` | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 575ce0b3c..27d021fc8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,7 +18,7 @@ embedded Postgres (PGlite), in-process events, and S3 to a local directory. | `bin/setup` | fresh clone to running | | `bin/dev ` | run the `x` CLI from source: `bin/dev verify --json` | | `bin/check` | the CI gate, locally | -| `bun run test` | every package's tests (bare `bun test` also picks up the opt-in suites — see #9) | +| `bun run test` | every package's tests, opt-in suites included; `examples/` is gated by its own `x verify` | | `bun run scripts/help.ts` | the full script catalogue | ## ✅ The gate @@ -26,17 +26,25 @@ embedded Postgres (PGlite), in-process events, and S3 to a local directory. **`bun run scripts/verify.ts` — green means shippable.** CI runs exactly this. There is no second checklist and no CI-only step. +It **is** `x verify`, run at the repo root: one step list, defined once in +`packages/cli/src/cmd-verify.ts`, so a contributor and a user see the same steps. A step that has +nothing to check here is reported as skipped (`-`), never as passed. + | Step | Fails on | |---|---| | `typecheck` | any type error; `any` is banned, and a cast is not a fix | | `lint` | Biome: formatting, `any`, unused, default exports | -| `boundaries` | a tier violation (see below) | -| `test` | any failing test; a flake is a failure | +| `boundaries` | a tier violation (see below) or an app surface violation | | `filesize` | a file over 500 lines | | `package-shape` | a package missing `README.md`, `CLAUDE.md`, `tsconfig.json`, `src/index.ts` | -| `manifest` | the framework manifest cannot be generated | - -Narrow it while iterating: `bun run scripts/verify.ts --only lint,boundaries --json`. +| `unit` | any failing test that is not one of the typed suites below; a flake is a failure | +| `contract` `live` `job` `e2e` `eval` | any failing `*..test.ts` suite (or any test under `e2e/`) | +| `drift` | an app schema that no migration recorded | +| `contract-diff` | a breaking change to a published action without a version bump | +| `budgets` | per-route JS bytes or LCP over the declared limit | +| `manifest` | the manifest differs from what the code produces, or cannot be generated | + +There is no `--only` and no `--skip`: "green" has to mean the same thing for everyone (axiom 5). ## 📦 Import tiers (a build error, not a preference) diff --git a/bun.lock b/bun.lock index 477d8027f..9df30df35 100644 --- a/bun.lock +++ b/bun.lock @@ -341,6 +341,9 @@ "version": "0.0.1", "dependencies": { "@ultimat3/core": "0.0.1", + "@ultimat3/jobs": "0.0.1", + "@ultimat3/mail": "0.0.1", + "@ultimat3/time": "0.0.1", }, }, "packages/time": { diff --git a/bunfig.toml b/bunfig.toml index 744f08658..13f6cb933 100644 --- a/bunfig.toml +++ b/bunfig.toml @@ -6,5 +6,8 @@ production = false [test] root = "." -# Frozen clock, seeded RNG, sealed network. See packages/testing. +# Frozen clock, seeded RNG, sealed network, framework fixtures. See packages/testing. +# Only the framework's preload: `examples/dummy` is a separate project with its own bunfig, +# its own fixtures (`seed`, `actorFor`) and its own gate. Registering the app's fixtures here +# would load the app's entity graph and mail registry into every framework test. preload = ["./scripts/test-setup.ts"] diff --git a/docs/architecture/14-testing-internals.md b/docs/architecture/14-testing-internals.md index e6331be88..461e73ad9 100644 --- a/docs/architecture/14-testing-internals.md +++ b/docs/architecture/14-testing-internals.md @@ -57,7 +57,7 @@ X_TEST_NETWORK_EGRESS: unmocked request in a test fix: http.mock('POST https://api.stripe.com/v1/charges', { status: 200, body: {...} }) ``` -The trap is installed at the fetch/socket layer, so it catches SDKs and transitive dependencies, not just direct `fetch` calls. There is no allowlist flag; a genuine integration test declares `network: 'live'` on the file and is excluded from `x verify`'s default set. +The trap is installed at the fetch/socket layer, so it catches SDKs and transitive dependencies, not just direct `fetch` calls. A server the test itself started is not egress — `start()` announces its socket to core, so a request back to that port passes. There is no allowlist API a file can call; the one opt-out is `ULTIMATE_TEST_ALLOW_NET=1` in the environment, reserved for a deliberate live integration, so no test can quietly unseal the network for itself. ## The six test types @@ -120,6 +120,8 @@ $ x verify | 1 | typecheck | any error; `any` is banned by lint, not tolerated by a cast | fastest signal, and everything downstream assumes types hold | | 2 | lint (Biome) | formatting, `any`, default exports, bare `Error`, raw hex, hardcoded strings | seconds, and it catches the cross-cutting rules before an expensive test run | | 3 | **import boundaries** | `site/` → `app/`, routes → DB, services → HTTP, framework tier violations | an import-scan pass; a boundary break invalidates the bundle-graph assumptions the later budget check depends on | +| 3a | file size | a source file over 500 lines | a file read; one file, one job is cheapest to check before anything runs | +| 3b | package shape | a workspace package missing `README.md`, `CLAUDE.md`, `tsconfig.json`, `src/index.ts` | four `stat` calls, and every later step assumes the package is navigable | | 4 | unit tests | any failure | no DB, so still cheap; fails fast on logic | | 5 | contract, live, job tests | any failure; a flake **is** a failure | needs cloned databases — first genuinely expensive step | | 6 | **migration drift** | schema ≠ migrations ≠ catalog, or an irreversible migration without a marker | after tests, because tests are what would have exercised the new column | diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 6e5e086b8..49cdcf61a 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -77,4 +77,4 @@ A convention that isn't a build error doesn't exist. What actually fails, and wh | SEO + i18n | missing meta, duplicate meta, missing key in any locale | [`09`](./09-rendering-internals.md), [`10`](./10-cross-cutting.md) | | manifest freshness | `x.manifest.json` / `openapi.json` differ from the code | [`11`](./11-ai-surface.md) | -One command runs all of them: `bun run verify` in this repo, `x verify` in a generated app. There is no `--skip`. +One command runs all of them, from one step list: `bun run verify` in this repo *is* `x verify` run at the repo root — same steps, same report, same exit code. There is no `--only` and no `--skip`. diff --git a/examples/dummy/CLAUDE.md b/examples/dummy/CLAUDE.md index c0580e7ae..03f9e0d5f 100644 --- a/examples/dummy/CLAUDE.md +++ b/examples/dummy/CLAUDE.md @@ -76,6 +76,9 @@ Feature slice: `apps/web/app//{entity,repo,service,actions,mutator,live - `idempotencyKey` on every job is required by the type. Keys derive from `input` only. - Tests sit next to their source: `.test.ts` (unit), `.contract.test.ts`, `.live.test.ts`, `.job.test.ts`, `.e2e.test.ts`, `.eval.test.ts`. +- Test fixtures come from `scripts/test-setup.ts`, the one preload in `bunfig.toml`. `clock`, + `mail` and `runJobs` are the framework's; `seed` and `actorFor` are Postly's. A test that + destructures anything else fails with `X_TEST_FIXTURE_UNKNOWN` — register it there, once. ## Boundaries (build errors, not lint warnings) diff --git a/examples/dummy/bunfig.toml b/examples/dummy/bunfig.toml new file mode 100644 index 000000000..73e0e7c0f --- /dev/null +++ b/examples/dummy/bunfig.toml @@ -0,0 +1,12 @@ +# Bun runtime + test config for Postly. https://bun.sh/docs/runtime/bunfig +# +# One preload, two jobs: `scripts/test-setup.ts` pulls in @ultimat3/testing's preload (frozen +# clock, seeded RNG, sealed network, matchers) and registers the fixtures the app owns. + +[install] +exact = false +production = false + +[test] +root = "." +preload = ["./scripts/test-setup.ts"] diff --git a/examples/dummy/packages/db/migrations/0001_init.hash b/examples/dummy/packages/db/migrations/0001_init.hash new file mode 100644 index 000000000..74aec7f51 --- /dev/null +++ b/examples/dummy/packages/db/migrations/0001_init.hash @@ -0,0 +1 @@ +81a4b14eea7ebdea diff --git a/examples/dummy/scripts/test-setup.ts b/examples/dummy/scripts/test-setup.ts new file mode 100644 index 000000000..137156928 --- /dev/null +++ b/examples/dummy/scripts/test-setup.ts @@ -0,0 +1,110 @@ +// Postly's test preload: the fixtures the APP owns. `clock`, `mail` and `runJobs` arrive with +// the framework's own preload, imported below — an app registers only what the framework cannot +// know, which here is the seed graph and how a member becomes an actor. +// +// [test] +// preload = ["./scripts/test-setup.ts"] +// +// The framework is imported by relative path rather than by `@ultimat3/*`: this directory is not +// a workspace member yet (issue #9), and a preload runs before anything else, so it must not +// depend on workspace symlinks. A generated app writes `@ultimat3/testing` here. For the same +// reason `scripts/` is not in tsconfig's `include` — a composite project cannot reach across +// into another one's sources. Both go away when the app joins the workspace. + +import { assert, userActor } from '../../../packages/core/src/index'; +import type { Driver, EntityCore, Repo, Seed } from '../../../packages/entity/src/index'; +import { memoryDriver, seedId } from '../../../packages/entity/src/index'; +import { defineFixtures } from '../../../packages/testing/src/index'; +import '../../../packages/testing/src/preload'; + +/** Every seeded row carries an id; the rest of the columns are the entity's business. */ +export interface SeedRow { + readonly id: string; + readonly [column: string]: unknown; +} + +export interface SeedHandle { + /** `pick({ draft: 'post:draft-money' })` — seed labels in, rows out, aliased at the call site. */ + pick>>( + labels: M, + ): Promise<{ readonly [K in keyof M]: SeedRow }>; +} + +const idOf = (row: unknown): string | undefined => { + const value = (row as { readonly id?: unknown }).id; + return typeof value === 'string' ? value : undefined; +}; + +/** + * Rows are captured on the way in rather than read back out: a tenant-scoped entity refuses an + * unscoped read, so a fixture would have to name the org before it could fetch the org. Insert + * still runs `$parse` and the invariants, so seeding still tests the schema. + */ +const capturingDriver = (rows: Map): Driver => { + const base = memoryDriver(); + return { + repo: (entity: EntityCore): Repo => { + const inner = base.repo(entity); + return { + ...inner, + insert: async (values, options) => { + const row = await inner.insert(values, options); + const id = idOf(row); + if (id !== undefined) rows.set(id, row as SeedRow); + return row; + }, + }; + }, + }; +}; + +/** A fresh graph per call: two tests must never see each other's writes. */ +const handleFor = (seed: Seed): SeedHandle => { + const rows = new Map(); + const ready = seed.run({ driver: capturingDriver(rows) }); + + return { + pick: async >>(labels: M) => { + await ready; + const picked: Record = {}; + for (const [alias, label] of Object.entries(labels)) { + const row = rows.get(seedId(label)); + assert( + row !== undefined, + `seed "${seed.name}" has no row labelled "${label}"`, + `add it to packages/db/seeds/${seed.name}.ts with id: id('${label}')`, + ); + picked[alias] = row; + } + return picked as { readonly [K in keyof M]: SeedRow }; + }, + }; +}; + +/** Imported on demand, so a test that never seeds never loads the entity graph. */ +const createSeed = async (): Promise<(name: string) => SeedHandle> => { + const { dev } = await import('../packages/db/seeds/dev'); + const seeds: Readonly> = { dev }; + return (name) => { + const seed = seeds[name]; + assert( + seed !== undefined, + `no seed named "${name}" — known seeds: ${Object.keys(seeds).join(', ')}`, + 'x db seed --list, then use one of the names it prints', + ); + return handleFor(seed); + }; +}; + +/** A member row is the actor: same org, and its membership role is the authz role. */ +const actorFor = (member: SeedRow) => + userActor({ + id: String(member['userId'] ?? member.id), + orgId: String(member['orgId']), + roles: [String(member['role'])], + }); + +defineFixtures({ + seed: createSeed, + actorFor: () => actorFor, +}); diff --git a/examples/dummy/tsconfig.json b/examples/dummy/tsconfig.json index 08eeb13b1..6ac4a0e5b 100644 --- a/examples/dummy/tsconfig.json +++ b/examples/dummy/tsconfig.json @@ -16,15 +16,16 @@ "skipLibCheck": true, "noEmit": true, "resolveJsonModule": true, - "baseUrl": ".", + "composite": true, + "declaration": true, "paths": { - "@postly/domain": ["packages/domain/src/index.ts"], - "@postly/db": ["packages/db/src/index.ts"], - "@postly/core": ["packages/core/src/index.ts"], - "@postly/i18n": ["packages/i18n/src/index.ts"], - "@postly/ui": ["packages/ui/src/index.ts"], - "@postly/mcp": ["packages/mcp/src/index.ts"], - "@postly/web/*": ["apps/web/*"] + "@postly/domain": ["./packages/domain/src/index.ts"], + "@postly/db": ["./packages/db/src/index.ts"], + "@postly/core": ["./packages/core/src/index.ts"], + "@postly/i18n": ["./packages/i18n/src/index.ts"], + "@postly/ui": ["./packages/ui/src/index.ts"], + "@postly/mcp": ["./packages/mcp/src/index.ts"], + "@postly/web/*": ["./apps/web/*"] } }, "include": ["app.config.ts", "apps/**/*", "packages/**/*"], diff --git a/package.json b/package.json index c09e84569..4c84d940a 100644 --- a/package.json +++ b/package.json @@ -15,8 +15,8 @@ "lint": "biome check .", "lint:fix": "biome check --write .", "format": "biome format --write .", - "test": "bun test --path-ignore-patterns='**/dist/**' --path-ignore-patterns='**/e2e/**' --path-ignore-patterns='**/*.{contract,live,job,eval}.test.ts'", - "test:watch": "bun test --watch --path-ignore-patterns='**/dist/**'", + "test": "bun test --path-ignore-patterns='**/dist/**' --path-ignore-patterns='**/examples/**'", + "test:watch": "bun test --watch --path-ignore-patterns='**/dist/**' --path-ignore-patterns='**/examples/**'", "verify": "bun run scripts/verify.ts", "boundaries": "bun run scripts/boundaries.ts", "manifest": "bun run scripts/manifest.ts", diff --git a/packages/cli/README.md b/packages/cli/README.md index 1f2f20b7f..174c018ea 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -9,7 +9,7 @@ The `x` binary. One char during dev, one command per job, `--json` on every one | `x new ` | scaffolds the monorepo | interactive-free; auth, seeded DB, example route | | `x dev` | every role in one process | embedded Postgres/events/storage, `/_x` mounted | | `x build --target docker\|binary\|static` | one artifact | `ROLE` selects behaviour at start | -| `x verify` | **the gate** | 13 named steps, each with pass/fail + duration | +| `x verify` | **the gate** | 15 named steps, each with pass/fail + duration | | `x g ` | scaffolds a primitive **with a passing test** | never a TODO stub | | `x db gen\|migrate\|reset\|studio\|branch` | everything DB | `branch` = copy-on-write clone + preview URL | | `x mcp serve` | MCP over stdio or HTTP | read tools unrestricted in dev | @@ -30,15 +30,18 @@ X_DB_DRIFT: schema differs from migrations ```sh x verify --json -# {"ok":false,"command":"verify","summary":"1 of 13 steps failed","steps":[...]} +# {"ok":false,"command":"verify","summary":"1 of 15 steps failed","steps":[...]} ``` ## `x verify` steps -`typecheck lint boundaries unit contract live job e2e eval drift contract-diff budgets manifest` +`typecheck lint boundaries filesize package-shape unit contract live job e2e eval drift +contract-diff budgets manifest` -Never bails early: an agent fixing three things needs all three findings from one run. -`--only a,b` and `--skip c` narrow it; the exit code is non-zero if any step fails. +One list, in cost order, defined once in `cmd-verify.ts` — the framework repo's own gate +(`bun run verify`) runs exactly it. A step with nothing to check here reports as skipped, never as +passed. Never bails early: an agent fixing three things needs all three findings from one run. +There is no `--only` and no `--skip`; the exit code is non-zero if any step fails. ## Layout @@ -52,6 +55,9 @@ Never bails early: an agent fixing three things needs all three findings from on | `cmd-*.ts` | one command group each | | `templates/` | scaffolding as typed string modules, not copied fixtures | | `surfaces.ts` | app import boundaries (site→app, shared leaf, route→DB) | +| `verify-step.ts` | the step shape, the step names, the host-check hook | +| `verify-tests.ts` | one `bun test` invocation per test type | +| `workspace-checks.ts` | file-size ceiling and package contract files | | `drift.ts` `openapi.ts` `budgets.ts` `manifest-scan.ts` | the checks `x verify` composes | ## Generated file layout diff --git a/packages/cli/src/cmd-build.ts b/packages/cli/src/cmd-build.ts index 43e5b34bb..776f8e1c4 100644 --- a/packages/cli/src/cmd-build.ts +++ b/packages/cli/src/cmd-build.ts @@ -81,8 +81,7 @@ export const buildCommand: CliCommand = { { code: 'X_BUILD_FAILED', cause: `${command.join(' ')} exited ${result.code}`, - fix: - target === 'docker' ? 'x doctor --json && docker info' : 'x verify --only typecheck', + fix: target === 'docker' ? 'x doctor --json && docker info' : 'x verify --json', docs: 'https://ultimate.dev/errors/X_BUILD_FAILED', }, ]; diff --git a/packages/cli/src/cmd-test.test.ts b/packages/cli/src/cmd-test.test.ts index cb3b53442..3d056b152 100644 --- a/packages/cli/src/cmd-test.test.ts +++ b/packages/cli/src/cmd-test.test.ts @@ -1,4 +1,5 @@ import { describe, expect, test } from 'bun:test'; +import { join } from 'node:path'; import type { Shard, TestFile } from './cmd-test'; import { discoverTests, planShards, reproduceFor, runShards, shardArgs } from './cmd-test'; import type { ExecOptions, Runner } from './exec'; @@ -189,7 +190,13 @@ describe('unit · x test discovery', () => { expect(files.every((file) => file.bytes > 0)).toBe(true); expect(files.every((file) => !file.path.includes('node_modules'))).toBe(true); expect(files.every((file) => !file.path.includes('/dist/'))).toBe(true); - expect(files.every((file) => !file.path.includes('/e2e/'))).toBe(true); + }); + + // `x test` and `bun run test` must see one suite. An e2e file dropped here is a file that only + // ever runs in CI, which is how `packages/http/e2e` stayed broken for as long as it did. + test('an opt-in e2e suite is discovered, not silently dropped', async () => { + const files = await discoverTests(join(import.meta.dir, '..', '..', 'http')); + expect(files.map((file) => file.path)).toContain('e2e/server.e2e.test.ts'); }); test('a filter narrows the set to matching paths', async () => { diff --git a/packages/cli/src/cmd-test.ts b/packages/cli/src/cmd-test.ts index 981a3f738..ec82ef377 100644 --- a/packages/cli/src/cmd-test.ts +++ b/packages/cli/src/cmd-test.ts @@ -28,8 +28,13 @@ export interface Shard { const TEST_GLOB = '**/*.test.ts'; -/** The root `test` script's ignore list, kept identical so `x test` and `bun test` see one suite. */ -const IGNORED = ['/dist/', '/node_modules/', '/e2e/']; +/** + * The root `test` script's ignore list, kept identical so `x test` and `bun run test` see one + * suite. `e2e/` is NOT on it: an opt-in suite that the gate runs but `x test` silently drops is + * a suite nobody runs until CI says so. `examples/` is, because the reference app is a separate + * project with its own gate — `x verify` there, not `x test` here. + */ +const IGNORED = ['/dist/', '/node_modules/', '/examples/']; /** File size stands in for duration: cheap to read, and it correlates far better than file count. */ export async function discoverTests(root: string, filter?: string): Promise { diff --git a/packages/cli/src/cmd-verify.test.ts b/packages/cli/src/cmd-verify.test.ts index ef0e3f4da..f25f0c5aa 100644 --- a/packages/cli/src/cmd-verify.test.ts +++ b/packages/cli/src/cmd-verify.test.ts @@ -1,9 +1,10 @@ import { describe, expect, test } from 'bun:test'; -import type { VerifyStep } from './cmd-verify'; -import { runVerify, VERIFY_STEPS, verifyStepNames } from './cmd-verify'; +import { runVerify, VERIFY_STEPS, verifyCommand, verifyStepNames } from './cmd-verify'; import { exitCodeFor } from './output'; +import type { VerifyContext, VerifyStep } from './verify-step'; +import { VERIFY_STEP_NAMES } from './verify-step'; -const ctx = { +const ctx: VerifyContext = { root: '/nowhere', runner: async () => ({ command: ['true'], @@ -45,6 +46,8 @@ describe('unit · x verify', () => { 'typecheck', 'lint', 'boundaries', + 'filesize', + 'package-shape', 'unit', 'contract', 'live', @@ -59,6 +62,10 @@ describe('unit · x verify', () => { expect(VERIFY_STEPS.every((step) => step.summary.length > 0)).toBe(true); }); + test('the declared names and the steps that exist are one list', () => { + expect(verifyStepNames()).toEqual([...VERIFY_STEP_NAMES]); + }); + test('a failing step makes the whole run fail and exit non-zero', async () => { const result = await runVerify(stubs, ctx); expect(result.ok).toBe(false); @@ -91,12 +98,34 @@ describe('unit · x verify', () => { expect(seen).toEqual(['typecheck', 'drift']); }); - test('--only runs one step and --skip removes one', async () => { - const only = await runVerify(stubs, ctx, { only: ['typecheck'] }); - expect(only.steps?.map((step) => step.name)).toEqual(['typecheck']); - expect(only.ok).toBe(true); - const skipped = await runVerify(stubs, ctx, { skip: ['drift'] }); - expect(skipped.ok).toBe(true); + test('there is no way to narrow the run: every step runs, every time', async () => { + const result = await runVerify(stubs, ctx); + expect(result.steps?.map((step) => step.name)).toEqual(['typecheck', 'drift', 'e2e']); + expect(verifyCommand.spec.flags).toEqual([]); + expect(verifyCommand.spec.usage).toBe('x verify [--json]'); + }); + + test('a host check adds findings to the step it was registered for', async () => { + const withHost: readonly VerifyStep[] = [ + { + name: 'boundaries', + summary: 'imports', + run: async (context) => { + const extra = (await context.hostChecks?.boundaries?.(context.root)) ?? []; + return { ok: extra.length === 0, findings: extra }; + }, + }, + ]; + const result = await runVerify(withHost, { + ...ctx, + hostChecks: { + boundaries: async () => [ + { code: 'X_BOUNDARY_VIOLATION', cause: 'cli imports admin', fix: 'invert the import' }, + ], + }, + }); + expect(result.ok).toBe(false); + expect(result.steps?.[0]?.findings[0]?.code).toBe('X_BOUNDARY_VIOLATION'); }); test('a step that throws becomes a finding, not a crash', async () => { @@ -112,6 +141,6 @@ describe('unit · x verify', () => { const result = await runVerify(boom, ctx); expect(result.ok).toBe(false); expect(result.steps?.[0]?.findings[0]?.code).toBe('X_VERIFY_FAILED'); - expect(result.steps?.[0]?.findings[0]?.fix).toBe('x verify --only boundaries --json'); + expect(result.steps?.[0]?.findings[0]?.fix).toBe('x verify --json'); }); }); diff --git a/packages/cli/src/cmd-verify.ts b/packages/cli/src/cmd-verify.ts index 1f1b26d89..8dda02b2e 100644 --- a/packages/cli/src/cmd-verify.ts +++ b/packages/cli/src/cmd-verify.ts @@ -1,80 +1,24 @@ // `x verify` — the contract. Every check is a named step with its own pass/fail and duration, the // same list in the terminal and in --json, and a non-zero exit if any step fails. Green means -// shippable (axiom 5); there is no second checklist and no CI-only step. +// shippable (axiom 5): one step list, no second checklist, no CI-only step, and no way to narrow +// the run — `--only` and `--skip` would make "green" mean whatever the caller chose. import { existsSync } from 'node:fs'; import { join } from 'node:path'; -import { requireAppRoot } from './app-root'; +import { APP_CONFIG_FILE, MANIFEST_FILE, requireAppRoot } from './app-root'; import { checkBudgets, readBuildStats } from './budgets'; import type { CliCommand, CommandContext } from './command'; import { checkDrift } from './drift'; -import type { ExecResult, Runner } from './exec'; -import { execOutput } from './exec'; import { scanApp } from './manifest-scan'; import { msg } from './messages'; import type { OpenApiDocument } from './openapi'; import { buildOpenApi, diffOpenApi } from './openapi'; import type { CommandResult, Finding, StepResult } from './output'; -import { flagList } from './parse'; import { checkAppBoundaries } from './surfaces'; - -export interface VerifyContext { - readonly root: string; - readonly runner: Runner; -} - -export interface StepOutcome { - readonly ok: boolean; - readonly findings: readonly Finding[]; - readonly output?: string; -} - -export interface VerifyStep { - readonly name: string; - readonly summary: string; - /** Returns false to record the step as skipped rather than passed. */ - applies?(ctx: VerifyContext): Promise; - run(ctx: VerifyContext): Promise; -} - -const passed: StepOutcome = { ok: true, findings: [] }; - -function fromExec(result: ExecResult, finding: Omit): StepOutcome { - if (result.ok) return { ok: true, findings: [], output: execOutput(result) }; - return { - ok: false, - findings: [{ ...finding, docs: `https://ultimate.dev/errors/${finding.code}` }], - output: execOutput(result), - }; -} - -const fromFindings = (findings: readonly Finding[]): StepOutcome => ({ - ok: findings.length === 0, - findings, -}); - -/** One `bun test` invocation per test type; the type helpers prefix their describe blocks. */ -function testStep(name: string, requires?: string): VerifyStep { - const step: VerifyStep = { - name, - summary: `${name} tests`, - async run(ctx) { - const result = await ctx.runner(['bun', 'test', '--test-name-pattern', `${name} · `], { - cwd: ctx.root, - }); - return fromExec(result, { - code: 'X_TEST_FAILED', - cause: `one or more ${name} tests failed`, - fix: `bun test --test-name-pattern "${name} · "`, - }); - }, - }; - if (requires === undefined) return step; - return { - ...step, - applies: async (ctx) => existsSync(join(ctx.root, requires)), - }; -} +import type { StepOutcome, VerifyContext, VerifyStep, VerifyStepName } from './verify-step'; +import { fromExec, fromFindings, hostFindings, passed } from './verify-step'; +import { TEST_STEPS } from './verify-tests'; +import { checkFileSizes, checkPackageShape, hasWorkspacePackages } from './workspace-checks'; const readOpenApi = async (root: string): Promise => { const path = join(root, 'openapi.json'); @@ -82,7 +26,7 @@ const readOpenApi = async (root: string): Promise = return (await Bun.file(path).json()) as OpenApiDocument; }; -/** The nine checks of the contract, expanded so each test type reports on its own line. */ +/** The whole contract, in cost order. Every check the framework knows how to make lives here. */ export const VERIFY_STEPS: readonly VerifyStep[] = [ { name: 'typecheck', @@ -112,18 +56,30 @@ export const VERIFY_STEPS: readonly VerifyStep[] = [ }, { name: 'boundaries', - summary: 'surface and layer imports', - run: async (ctx) => fromFindings(await checkAppBoundaries(ctx.root)), + summary: 'surface, layer and package-tier imports', + run: async (ctx) => + fromFindings([ + ...(await checkAppBoundaries(ctx.root)), + ...(await hostFindings(ctx, 'boundaries')), + ]), }, - testStep('unit'), - testStep('contract'), - testStep('live'), - testStep('job'), - { ...testStep('e2e', 'e2e'), summary: 'playwright, incl. offline + SW update' }, - { ...testStep('eval', 'evals'), summary: 'LLM output scoring against thresholds' }, + { + name: 'filesize', + summary: 'one file, one job', + run: async (ctx) => fromFindings(await checkFileSizes(ctx.root)), + }, + { + name: 'package-shape', + summary: 'every package ships the same contract files', + applies: (ctx) => hasWorkspacePackages(ctx.root), + run: async (ctx) => fromFindings(await checkPackageShape(ctx.root)), + }, + ...TEST_STEPS, { name: 'drift', summary: 'schema vs migrations', + // Only an app owns migrations; a package monorepo's `packages/db` is the driver, not a schema. + applies: async (ctx) => existsSync(join(ctx.root, APP_CONFIG_FILE)), run: async (ctx) => fromFindings(await checkDrift(ctx.root)), }, { @@ -150,52 +106,45 @@ export const VERIFY_STEPS: readonly VerifyStep[] = [ }, { name: 'manifest', - summary: 'x.manifest.json freshness', - applies: async (ctx) => existsSync(join(ctx.root, 'x.manifest.json')), + summary: 'the generated manifest matches the code', + applies: async (ctx) => + existsSync(join(ctx.root, MANIFEST_FILE)) || ctx.hostChecks?.manifest !== undefined, async run(ctx) { - const committed = (await Bun.file(join(ctx.root, 'x.manifest.json')).json()) as { - buildId?: string; - }; - const current = await scanApp({ root: ctx.root }); - if (committed.buildId === current.buildId) return passed; return fromFindings([ - { - code: 'X_MANIFEST_STALE', - cause: `x.manifest.json records build ${committed.buildId ?? 'none'}, the code produces ${current.buildId}`, - fix: 'x manifest', - docs: 'https://ultimate.dev/errors/X_MANIFEST_STALE', - at: 'x.manifest.json', - }, + ...(await appManifestFindings(ctx.root)), + ...(await hostFindings(ctx, 'manifest')), ]); }, }, ]; -export interface VerifyOptions { - readonly only?: readonly string[]; - readonly skip?: readonly string[]; +async function appManifestFindings(root: string): Promise { + const path = join(root, MANIFEST_FILE); + if (!existsSync(path)) return []; + const committed = (await Bun.file(path).json()) as { buildId?: string }; + const current = await scanApp({ root }); + if (committed.buildId === current.buildId) return []; + return [ + { + code: 'X_MANIFEST_STALE', + cause: `${MANIFEST_FILE} records build ${committed.buildId ?? 'none'}, the code produces ${current.buildId}`, + fix: 'x manifest', + docs: 'https://ultimate.dev/errors/X_MANIFEST_STALE', + at: MANIFEST_FILE, + }, + ]; } -const selected = (steps: readonly VerifyStep[], options: VerifyOptions): readonly VerifyStep[] => { - const only = options.only ?? []; - const skip = options.skip ?? []; - return steps.filter( - (step) => (only.length === 0 || only.includes(step.name)) && !skip.includes(step.name), - ); -}; - /** - * Run steps in order, never bailing early: an agent fixing three things at once needs all three - * findings from one run, not one per round-trip. + * Run every step in order, never bailing early: an agent fixing three things at once needs all + * three findings from one run, not one per round-trip. */ export async function runVerify( steps: readonly VerifyStep[], ctx: VerifyContext, - options: VerifyOptions = {}, ): Promise { - const chosen = selected(steps, options); const results: StepResult[] = []; - for (const step of chosen) { + for (const step of steps) { const applies = step.applies === undefined ? true : await step.applies(ctx); if (!applies) { results.push({ name: step.name, ok: true, durationMs: 0, skipped: true, findings: [] }); @@ -236,7 +185,7 @@ function findingOf(error: unknown, step: string): Finding { return { code: 'X_VERIFY_FAILED', cause: `step "${step}" threw: ${cause}`, - fix: `x verify --only ${step} --json`, + fix: 'x verify --json', docs: 'https://ultimate.dev/errors/X_VERIFY_FAILED', }; } @@ -245,21 +194,15 @@ export const verifyCommand: CliCommand = { spec: { name: 'verify', summary: 'the gate: typecheck, lint, boundaries, all tests, drift, contract, budgets', - usage: 'x verify [--only step,step] [--skip step] [--json]', + usage: 'x verify [--json]', requiresApp: true, - flags: [ - { name: 'only', type: 'string', summary: 'comma-separated step names to run' }, - { name: 'skip', type: 'string', summary: 'comma-separated step names to skip' }, - ], + flags: [], }, async run(ctx: CommandContext): Promise { const root = requireAppRoot('verify', ctx.cwd).dir; - return runVerify( - VERIFY_STEPS, - { root, runner: ctx.runner }, - { only: flagList(ctx.args, 'only'), skip: flagList(ctx.args, 'skip') }, - ); + return runVerify(VERIFY_STEPS, { root, runner: ctx.runner }); }, }; -export const verifyStepNames = (): readonly string[] => VERIFY_STEPS.map((step) => step.name); +export const verifyStepNames = (): readonly VerifyStepName[] => + VERIFY_STEPS.map((step) => step.name); diff --git a/packages/cli/src/errors.ts b/packages/cli/src/errors.ts index 901901900..203b59975 100644 --- a/packages/cli/src/errors.ts +++ b/packages/cli/src/errors.ts @@ -48,7 +48,7 @@ export class VerifyFailedError extends UltimateError { super({ code: 'X_VERIFY_FAILED', cause: `${input.failed.length} verify step(s) failed: ${input.failed.join(', ')}`, - fix: `x verify --only ${input.failed[0] ?? 'typecheck'} --json`, + fix: 'x verify --json', docs: docsFor('X_VERIFY_FAILED'), }); } diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 8cc473984..fcdbadf26 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -40,7 +40,6 @@ export { shardArgs, testCommand, } from './cmd-test'; -export type { StepOutcome, VerifyContext, VerifyOptions, VerifyStep } from './cmd-verify'; export { runVerify, VERIFY_STEPS, verifyCommand, verifyStepNames } from './cmd-verify'; export type { CliCommand, CommandContext } from './command'; export { failed, ok } from './command'; @@ -83,3 +82,21 @@ export { flagBool, flagList, flagString, GLOBAL_FLAGS, nearest, parseArgs } from export { CLI_VERSION, COMMANDS, commandFor, SPECS } from './registry'; export type { SourceFile, Surface as AppSurface } from './surfaces'; export { BOUNDARY_CODES, checkAppBoundaries, checkSurfaceRules, surfaceOf } from './surfaces'; +export type { + HostCheck, + StepOutcome, + VerifyContext, + VerifyStep, + VerifyStepName, +} from './verify-step'; +export { VERIFY_STEP_NAMES } from './verify-step'; +export type { TestType } from './verify-tests'; +export { TEST_STEPS, TEST_TYPES, testStepCommand } from './verify-tests'; +export { + checkFileSizes, + checkPackageShape, + hasWorkspacePackages, + LINE_CEILING, + PACKAGE_FILES, + workspacePackages, +} from './workspace-checks'; diff --git a/packages/cli/src/output.test.ts b/packages/cli/src/output.test.ts index fbf23236a..747570931 100644 --- a/packages/cli/src/output.test.ts +++ b/packages/cli/src/output.test.ts @@ -61,6 +61,23 @@ describe('unit · output', () => { expect(payload.steps[1]?.findings[0]?.fix).toBe('x db gen "add publish_at"'); }); + // `runVerify` sets `skipped` only on a step that does not apply, so an executed step reaches + // here without the key. A consumer parsing --json must not have to tell "ran" from "absent", + // which is why the render normalises it — and why the documented shape says `"skipped":false`. + test('every step in the JSON render carries an explicit skipped boolean', () => { + const payload = JSON.parse( + renderJson({ + ...failing, + steps: [ + { name: 'typecheck', ok: true, durationMs: 12, findings: [] }, + { name: 'e2e', ok: true, durationMs: 0, skipped: true, findings: [] }, + ], + }), + ) as { steps: { name: string; skipped: boolean }[] }; + expect(payload.steps.map((step) => step.skipped)).toEqual([false, true]); + expect(renderJson(failing)).toContain('"skipped":false'); + }); + test('an UltimateError-shaped value is recognised across a process boundary', () => { const plain = { code: 'X_TEST', cause: 'because', fix: 'x doctor' }; expect(isUltimateErrorShape(plain)).toBe(true); diff --git a/packages/cli/src/surfaces.ts b/packages/cli/src/surfaces.ts index 6b72e45f1..8bfd8c720 100644 --- a/packages/cli/src/surfaces.ts +++ b/packages/cli/src/surfaces.ts @@ -85,7 +85,7 @@ export function checkSurfaceRules(files: readonly SourceFile[]): readonly Findin violation('X_BOUNDARY_SITE_TO_APP', { at: file.path, cause: `site/ imports "${specifier}" from app/ — the marketing bundle would inherit the app graph`, - fix: `move the shared part into shared/ and import it from both, then: x verify --only boundaries`, + fix: `move the shared part into shared/ and import it from both, then: x verify --json`, }), ); } diff --git a/packages/cli/src/verify-step.ts b/packages/cli/src/verify-step.ts new file mode 100644 index 000000000..8687ab226 --- /dev/null +++ b/packages/cli/src/verify-step.ts @@ -0,0 +1,81 @@ +// The shape of one gate step: its name, what it checks, whether it applies here, and how a host +// repo feeds it findings it could not produce on its own. Split from the step list so a step +// implementation can live beside the code it checks without importing the list. + +import type { ExecResult, Runner } from './exec'; +import { execOutput } from './exec'; +import type { Finding } from './output'; + +/** + * Every step of the gate, in cost order — cheapest and most informative first, and never a check + * whose result would be meaningless because an earlier one failed. This list is the definition of + * shippable: the framework repo and a generated app run exactly it, whole, or not at all. + */ +export const VERIFY_STEP_NAMES = [ + 'typecheck', + 'lint', + 'boundaries', + 'filesize', + 'package-shape', + 'unit', + 'contract', + 'live', + 'job', + 'e2e', + 'eval', + 'drift', + 'contract-diff', + 'budgets', + 'manifest', +] as const; + +export type VerifyStepName = (typeof VERIFY_STEP_NAMES)[number]; + +/** + * A rule the host repo enforces inside an existing step — the framework monorepo's package tier + * table under `boundaries`, its generated manifest under `manifest`. A host adds findings to a + * step; it can never add, remove, reorder or skip one. + */ +export type HostCheck = (root: string) => Promise; + +export interface VerifyContext { + readonly root: string; + readonly runner: Runner; + readonly hostChecks?: Partial>; +} + +export interface StepOutcome { + readonly ok: boolean; + readonly findings: readonly Finding[]; + readonly output?: string; +} + +export interface VerifyStep { + readonly name: VerifyStepName; + readonly summary: string; + /** Returns false to record the step as skipped rather than passed. */ + applies?(ctx: VerifyContext): Promise; + run(ctx: VerifyContext): Promise; +} + +export const passed: StepOutcome = { ok: true, findings: [] }; + +export function fromExec(result: ExecResult, finding: Omit): StepOutcome { + if (result.ok) return { ok: true, findings: [], output: execOutput(result) }; + return { + ok: false, + findings: [{ ...finding, docs: `https://ultimate.dev/errors/${finding.code}` }], + output: execOutput(result), + }; +} + +export const fromFindings = (findings: readonly Finding[]): StepOutcome => ({ + ok: findings.length === 0, + findings, +}); + +/** What the host repo contributes to this step, or nothing. */ +export const hostFindings = async ( + ctx: VerifyContext, + step: VerifyStepName, +): Promise => (await ctx.hostChecks?.[step]?.(ctx.root)) ?? []; diff --git a/packages/cli/src/verify-tests.test.ts b/packages/cli/src/verify-tests.test.ts new file mode 100644 index 000000000..1118092e8 --- /dev/null +++ b/packages/cli/src/verify-tests.test.ts @@ -0,0 +1,103 @@ +import { describe, expect, test } from 'bun:test'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { VerifyContext } from './verify-step'; +import { TEST_STEPS, TEST_TYPES, testStepCommand } from './verify-tests'; + +const REPO_ROOT = new URL('../../..', import.meta.url).pathname.replace(/\/$/, ''); + +const ctxFor = (root: string): VerifyContext => ({ + root, + runner: async () => ({ + command: ['bun', 'test'], + code: 0, + ok: true, + stdout: '', + stderr: '', + durationMs: 0, + }), +}); + +describe('unit · one bun test invocation per test type', () => { + test('every test type is a step, in cost order', () => { + expect(TEST_STEPS.map((step) => step.name)).toEqual([...TEST_TYPES]); + }); + + test('unit claims everything the typed suites do not', () => { + const command = testStepCommand('unit').join(' '); + expect(command).toContain('--path-ignore-patterns=**/*.{contract,live,job,e2e,eval}.test.*'); + expect(command).toContain('--path-ignore-patterns=**/e2e/**'); + expect(command).toContain('--path-ignore-patterns=**/dist/**'); + }); + + test('a typed step selects its suffix and nothing else', () => { + expect(testStepCommand('contract')).toEqual([ + 'bun', + 'test', + '--path-ignore-patterns=**/dist/**', + '--path-ignore-patterns=**/build/**', + '--path-ignore-patterns=**/examples/**', + '.contract.test.', + ]); + expect(testStepCommand('job').at(-1)).toBe('.job.test.'); + expect(testStepCommand('e2e').at(-1)).toBe('e2e'); + }); + + // A nested project with its own `x verify` is gated once, by its own run. Collecting it here + // too would report the app's failures on the framework's gate and the app's gate both. + test('every type skips nested projects that carry their own gate', () => { + for (const type of TEST_TYPES) { + expect(testStepCommand(type)).toContain('--path-ignore-patterns=**/examples/**'); + } + }); + + test('a failed step tells you the exact command that reproduces it', async () => { + const step = TEST_STEPS.find((entry) => entry.name === 'job'); + const outcome = await step?.run({ + ...ctxFor(REPO_ROOT), + runner: async (command) => ({ + command, + code: 1, + ok: false, + stdout: '', + stderr: 'boom', + durationMs: 1, + }), + }); + expect(outcome?.ok).toBe(false); + expect(outcome?.findings[0]?.code).toBe('X_TEST_FAILED'); + expect(outcome?.findings[0]?.fix).toBe(testStepCommand('job').join(' ')); + }); + + test('a type with no suites here is skipped, never silently passed', async () => { + const empty = await mkdtemp(join(tmpdir(), 'ultimate-verify-tests-')); + try { + await Bun.write(join(empty, 'packages/core/src/core.test.ts'), 'export {};\n'); + const applies = async (name: string, root: string): Promise => + TEST_STEPS.find((step) => step.name === name)?.applies?.(ctxFor(root)); + expect(await applies('contract', empty)).toBe(false); + expect(await applies('e2e', empty)).toBe(false); + expect(await applies('e2e', REPO_ROOT)).toBe(true); + expect(TEST_STEPS.find((step) => step.name === 'unit')?.applies).toBeUndefined(); + } finally { + await rm(empty, { recursive: true, force: true }); + } + }); + + // `applies` and the command must read the same exclusions. When they disagreed, a suite that + // lived only under an ignored path made its step apply and then fail on "no test files matched". + test('a suite that only exists in an ignored path does not make its step apply', async () => { + const nested = await mkdtemp(join(tmpdir(), 'ultimate-verify-nested-')); + try { + await Bun.write(join(nested, 'examples/dummy/app/posts.contract.test.ts'), 'export {};\n'); + await Bun.write(join(nested, 'packages/cli/dist/bundled.e2e.test.ts'), 'export {};\n'); + const applies = async (name: string): Promise => + TEST_STEPS.find((step) => step.name === name)?.applies?.(ctxFor(nested)); + expect(await applies('contract')).toBe(false); + expect(await applies('e2e')).toBe(false); + } finally { + await rm(nested, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/cli/src/verify-tests.ts b/packages/cli/src/verify-tests.ts new file mode 100644 index 000000000..200f7bca2 --- /dev/null +++ b/packages/cli/src/verify-tests.ts @@ -0,0 +1,119 @@ +// One `bun test` invocation per test type, so every type reports on its own line of the gate. A +// test's type is its filename suffix — `*.contract.test.ts`, `*.live.test.ts`, `*.job.test.ts`, +// `*.e2e.test.ts` (or any file under an `e2e/` directory), `*.eval.test.ts`. Everything else is a +// unit test, which is why the unit step is the only one that selects by exclusion. + +import type { StepOutcome, VerifyContext, VerifyStep } from './verify-step'; +import { fromExec } from './verify-step'; + +export const TEST_TYPES = ['unit', 'contract', 'live', 'job', 'e2e', 'eval'] as const; + +export type TestType = (typeof TEST_TYPES)[number]; + +interface TestSuite { + readonly summary: string; + /** Substring `bun test` matches against each file path. */ + readonly filter: string; + /** Globs that decide whether this type exists here at all. */ + readonly globs: readonly string[]; +} + +const TYPED_SUFFIXES = '{contract,live,job,e2e,eval}'; + +const SUITES: Readonly, TestSuite>> = { + contract: { + summary: 'action/query schemas, policy denials, emitted OpenAPI and MCP shapes', + filter: '.contract.test.', + globs: ['**/*.contract.test.{ts,tsx}'], + }, + live: { + summary: 'live-query snapshots, incremental patches, reconnect deltas', + filter: '.live.test.', + globs: ['**/*.live.test.{ts,tsx}'], + }, + job: { + summary: 'step replay, idempotency dedupe, retry/backoff, outbox atomicity', + filter: '.job.test.', + globs: ['**/*.job.test.{ts,tsx}'], + }, + e2e: { + summary: 'the built output, incl. offline and SW update', + filter: 'e2e', + globs: ['**/*.e2e.test.{ts,tsx}', '**/e2e/**/*.test.{ts,tsx}'], + }, + eval: { + summary: 'LLM output scored against thresholds', + filter: '.eval.test.', + globs: ['**/*.eval.test.{ts,tsx}'], + }, +}; + +/** + * Build output, and nested projects that carry their own `x verify`. `examples/**` is the second + * kind: the reference app is gated by its own run of this same step list, so collecting it here + * would report one app failure on two different gates. The patterns are relative to the run's + * root, so this excludes nothing when the app itself is the root. + */ +const NEVER_A_TEST = ['**/dist/**', '**/build/**', '**/examples/**']; + +const ignoreFlags = (patterns: readonly string[]): readonly string[] => + patterns.map((pattern) => `--path-ignore-patterns=${pattern}`); + +/** Unit is everything the typed suites do not claim, so no test can fall between two steps. */ +export const testStepCommand = (type: TestType): readonly string[] => + type === 'unit' + ? [ + 'bun', + 'test', + ...ignoreFlags([...NEVER_A_TEST, '**/e2e/**', `**/*.${TYPED_SUFFIXES}.test.*`]), + ] + : ['bun', 'test', ...ignoreFlags(NEVER_A_TEST), SUITES[type].filter]; + +/** + * Whether a step applies has to be decided by the same rule that decides what it runs. When the + * two drifted, a suite that lived only under an ignored path made its step apply and then fail + * with "no test files matched" — a red gate reporting a suite that, by its own rule, is not here. + */ +const NEVER_A_TEST_GLOBS = NEVER_A_TEST.map((pattern) => new Bun.Glob(pattern)); + +const ignoredPath = (path: string): boolean => + path.includes('node_modules') || NEVER_A_TEST_GLOBS.some((glob) => glob.match(path)); + +const exists = async (root: string, globs: readonly string[]): Promise => { + for (const pattern of globs) { + for await (const path of new Bun.Glob(pattern).scan({ cwd: root, absolute: false })) { + if (!ignoredPath(path)) return true; + } + } + return false; +}; + +const runType = async (ctx: VerifyContext, type: TestType): Promise => { + const command = testStepCommand(type); + const result = await ctx.runner(command, { cwd: ctx.root }); + return fromExec(result, { + code: 'X_TEST_FAILED', + cause: `one or more ${type} tests failed`, + fix: command.join(' '), + }); +}; + +const stepFor = (type: TestType): VerifyStep => { + if (type === 'unit') { + return { + name: 'unit', + summary: 'pure logic — no database, no network', + run: (ctx) => runType(ctx, 'unit'), + }; + } + const suite = SUITES[type]; + return { + name: type, + summary: suite.summary, + applies: (ctx) => exists(ctx.root, suite.globs), + run: (ctx) => runType(ctx, type), + }; +}; + +/** In cost order: unit needs nothing, e2e needs a build. */ +export const TEST_STEPS: readonly VerifyStep[] = TEST_TYPES.map(stepFor); diff --git a/packages/cli/src/workspace-checks.test.ts b/packages/cli/src/workspace-checks.test.ts new file mode 100644 index 000000000..e8c457af5 --- /dev/null +++ b/packages/cli/src/workspace-checks.test.ts @@ -0,0 +1,107 @@ +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + checkFileSizes, + checkPackageShape, + countLines, + hasWorkspacePackages, + LINE_CEILING, + PACKAGE_FILES, + workspacePackages, +} from './workspace-checks'; + +const REPO_ROOT = new URL('../../..', import.meta.url).pathname.replace(/\/$/, ''); + +let dir = ''; + +/** Terminated by a newline, like every file Biome formats — the case the count has to get right. */ +const lines = (count: number): string => + `${Array.from({ length: count }, () => 'const x = 1;').join('\n')}\n`; + +beforeAll(async () => { + dir = await mkdtemp(join(tmpdir(), 'ultimate-workspace-checks-')); + await Bun.write(join(dir, 'packages/short/src/index.ts'), lines(LINE_CEILING)); + await Bun.write(join(dir, 'packages/short/README.md'), '# short\n'); + await Bun.write(join(dir, 'packages/short/CLAUDE.md'), '# short\n'); + await Bun.write(join(dir, 'packages/short/tsconfig.json'), '{}\n'); + await Bun.write(join(dir, 'packages/short/package.json'), '{"name":"short"}\n'); + await Bun.write(join(dir, 'packages/long/src/index.ts'), lines(LINE_CEILING + 1)); + await Bun.write(join(dir, 'packages/long/package.json'), '{"name":"long"}\n'); + await Bun.write(join(dir, 'packages/long/node_modules/dep/src/huge.ts'), lines(2_000)); + await Bun.write(join(dir, 'app/orgs/page.tsx'), lines(LINE_CEILING + 40)); +}); + +afterAll(async () => { + await rm(dir, { recursive: true, force: true }); +}); + +describe('unit · the file-size ceiling', () => { + test('one line over the ceiling is a finding, exactly at it is not', async () => { + const findings = await checkFileSizes(dir); + const paths = findings.map((finding) => finding.at); + expect(paths).toContain('packages/long/src/index.ts'); + expect(paths).not.toContain('packages/short/src/index.ts'); + expect(findings.every((finding) => finding.code === 'X_FILE_TOO_LONG')).toBe(true); + }); + + // The ceiling is 500, so it has to be 500 — off by one it enforces 499 and every count it + // reports is a line too high, which sends an author to split a file that already fits. + test('a terminating newline is not a line of its own', () => { + expect(countLines('')).toBe(0); + expect(countLines('a\n')).toBe(1); + expect(countLines('a')).toBe(1); + expect(countLines('a\nb\n')).toBe(2); + expect(countLines('a\nb')).toBe(2); + expect(countLines('a\n\n')).toBe(2); + expect(countLines(lines(LINE_CEILING))).toBe(LINE_CEILING); + }); + + test('a file exactly at the ceiling is counted at the ceiling, not over it', async () => { + const at = join(dir, 'packages/edge/src/index.ts'); + await Bun.write(at, lines(LINE_CEILING)); + await Bun.write(join(dir, 'packages/edge/src/over.ts'), lines(LINE_CEILING + 1)); + + const findings = await checkFileSizes(dir); + const over = findings.find((finding) => finding.at === 'packages/edge/src/over.ts'); + + expect(findings.map((finding) => finding.at)).not.toContain('packages/edge/src/index.ts'); + expect(over?.cause).toContain(`${LINE_CEILING + 1} lines`); + }); + + test('app surfaces are scanned too, and dependencies are not', async () => { + const paths = (await checkFileSizes(dir)).map((finding) => finding.at); + expect(paths).toContain('app/orgs/page.tsx'); + expect(paths.some((path) => path?.includes('node_modules'))).toBe(false); + }); + + test('every finding names the file it is about in its fix', async () => { + for (const finding of await checkFileSizes(dir)) { + expect(finding.fix).toContain(finding.at ?? ''); + } + }); +}); + +describe('unit · the package shape', () => { + test('a package missing a contract file is reported once per file', async () => { + const findings = await checkPackageShape(dir); + expect(findings.map((finding) => finding.at)).toEqual([ + 'packages/long/README.md', + 'packages/long/CLAUDE.md', + 'packages/long/tsconfig.json', + ]); + expect(findings.every((finding) => finding.code === 'X_PACKAGE_SHAPE')).toBe(true); + }); + + test('this repo satisfies the shape it enforces', async () => { + expect(await checkPackageShape(REPO_ROOT)).toEqual([]); + expect(await workspacePackages(REPO_ROOT)).toContain('cli'); + expect(PACKAGE_FILES).toHaveLength(4); + }); + + test('the step is skipped where there are no workspace packages', async () => { + expect(await hasWorkspacePackages(dir)).toBe(true); + expect(await hasWorkspacePackages(join(dir, 'app'))).toBe(false); + }); +}); diff --git a/packages/cli/src/workspace-checks.ts b/packages/cli/src/workspace-checks.ts new file mode 100644 index 000000000..312d9c1c0 --- /dev/null +++ b/packages/cli/src/workspace-checks.ts @@ -0,0 +1,99 @@ +// Two shape rules the gate owns: one file, one job (a hard line ceiling), and every workspace +// package shipping the same contract files. Both report findings — a shape rule that is only +// written down is not a rule (axiom 3). + +import { existsSync } from 'node:fs'; +import { join } from 'node:path'; +import type { Finding } from './output'; + +export const LINE_CEILING = 500; + +export const PACKAGE_FILES = ['README.md', 'CLAUDE.md', 'tsconfig.json', 'src/index.ts'] as const; + +/** + * Where source lives in both shapes this gate runs against: a package monorepo and an app. A + * nested example app under `examples/` is not scanned — it runs this same gate from its own root. + */ +const SOURCE_GLOBS = [ + 'packages/*/src/**/*.{ts,tsx}', + 'scripts/**/*.{ts,tsx}', + 'site/**/*.{ts,tsx}', + 'app/**/*.{ts,tsx}', + 'api/**/*.{ts,tsx}', + 'shared/**/*.{ts,tsx}', + 'apps/*/{app,site,api,shared}/**/*.{ts,tsx}', +] as const; + +const docs = (code: string): string => `https://ultimate.dev/errors/${code}`; + +const skip = (path: string): boolean => + path.includes('node_modules') || path.includes('/dist/') || path.startsWith('dist/'); + +export const tooLongFinding = (path: string, lines: number): Finding => ({ + code: 'X_FILE_TOO_LONG', + cause: `${path} is ${lines} lines, over the ${LINE_CEILING} line ceiling`, + fix: `split ${path}: one file, one responsibility`, + docs: docs('X_FILE_TOO_LONG'), + at: path, +}); + +/** + * A trailing newline terminates the last line, it does not start another one. Counting the split + * parts instead made the real ceiling 499 and reported every count one too high — every correctly + * formatted file here ends with a newline, which is exactly the case that was wrong. + */ +export const countLines = (text: string): number => + text === '' ? 0 : text.split('\n').length - (text.endsWith('\n') ? 1 : 0); + +/** Files are the unit of review: one file, one job, hard ceiling 500 lines. */ +export async function checkFileSizes(root: string): Promise { + const findings: Finding[] = []; + const seen = new Set(); + for (const pattern of SOURCE_GLOBS) { + for await (const path of new Bun.Glob(pattern).scan({ cwd: root, absolute: false })) { + if (skip(path) || seen.has(path)) continue; + seen.add(path); + const lines = countLines(await Bun.file(join(root, path)).text()); + if (lines > LINE_CEILING) findings.push(tooLongFinding(path, lines)); + } + } + return findings; +} + +export const missingFileFinding = (dir: string, file: string, scaffolder: boolean): Finding => ({ + code: 'X_PACKAGE_SHAPE', + cause: `packages/${dir} has no ${file}`, + fix: scaffolder + ? `bun run scripts/new-package.ts ${dir} --only ${file}` + : `add packages/${dir}/${file}, shaped like the one in a sibling package`, + docs: docs('X_PACKAGE_SHAPE'), + at: `packages/${dir}/${file}`, +}); + +export async function workspacePackages(root: string): Promise { + const dirs: string[] = []; + for await (const path of new Bun.Glob('packages/*/package.json').scan({ + cwd: root, + absolute: false, + })) { + const dir = path.split('/')[1]; + if (dir !== undefined) dirs.push(dir); + } + return dirs.sort(); +} + +export const hasWorkspacePackages = async (root: string): Promise => + (await workspacePackages(root)).length > 0; + +/** Every package ships the same contract files; a missing one is a build error, not a chore. */ +export async function checkPackageShape(root: string): Promise { + const scaffolder = existsSync(join(root, 'scripts', 'new-package.ts')); + const findings: Finding[] = []; + for (const dir of await workspacePackages(root)) { + for (const file of PACKAGE_FILES) { + if (existsSync(join(root, 'packages', dir, file))) continue; + findings.push(missingFileFinding(dir, file, scaffolder)); + } + } + return findings; +} diff --git a/packages/core/CLAUDE.md b/packages/core/CLAUDE.md index e1eeae905..d7439b6f4 100644 --- a/packages/core/CLAUDE.md +++ b/packages/core/CLAUDE.md @@ -29,4 +29,7 @@ Gotchas: - `exactOptionalPropertyTypes` is on — declare optional fields as `x?: T | undefined`. - `noPropertyAccessFromIndexSignature` is on — `ctx.services['mail']`, not `.mail`. - `Ctx` carries a string index signature so apps can augment `CtxServices` for `ctx.posts`. -- Tests that touch the registry or lifecycle must call `resetErrorCodes()` / `resetLifecycle()`. +- Tests that touch the registry, the lifecycle or the listener table must call + `resetErrorCodes()` / `resetLifecycle()` / `resetListeners()`. +- Anything that opens a socket calls `markListening(server.url.origin)` and releases it on close. + That is what tells the sealed test network a loopback request is this process, not egress. diff --git a/packages/core/README.md b/packages/core/README.md index 85d08f0c4..521720861 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -18,6 +18,7 @@ Zero dependencies, zero `@ultimat3/*` imports. | structured JSON logging + redaction | `logger.ts` | | OTel-shaped spans, always on, no-op by default | `telemetry.ts` | | graceful drain, `/healthz`, `/readyz` | `lifecycle.ts` | +| the sockets this process opened, so a self-request is not egress | `listeners.ts` | | `assertNever`, `invariant` | `assert.ts` | ## Errors are instructions diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 862db8d27..51f61c5f8 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -142,6 +142,12 @@ export { resetLifecycle, SHUTDOWN_PHASES, } from './lifecycle'; +export { + isSelfOrigin, + listeningOrigins, + markListening, + resetListeners, +} from './listeners'; export type { LogFields, Logger, LoggerOptions, LogLevel } from './logger'; export { createLogger, diff --git a/packages/core/src/listeners.test.ts b/packages/core/src/listeners.test.ts new file mode 100644 index 000000000..414942951 --- /dev/null +++ b/packages/core/src/listeners.test.ts @@ -0,0 +1,56 @@ +import { afterEach, describe, expect, test } from 'bun:test'; +import { isSelfOrigin, listeningOrigins, markListening, resetListeners } from './listeners'; + +afterEach(() => { + resetListeners(); +}); + +describe('listeners', () => { + test('an announced socket is self, an unannounced one is not', () => { + expect(isSelfOrigin('http://127.0.0.1:4310/healthz')).toBe(false); + markListening('http://127.0.0.1:4310'); + expect(isSelfOrigin('http://127.0.0.1:4310/healthz')).toBe(true); + expect(listeningOrigins()).toEqual(['http://127.0.0.1:4310']); + }); + + test('every loopback spelling of the same port is the same socket', () => { + markListening('http://0.0.0.0:4311'); + for (const url of [ + 'http://localhost:4311/x', + 'http://127.0.0.1:4311/x', + 'http://[::1]:4311/x', + 'http://LOCALHOST:4311/x', + ]) { + expect(isSelfOrigin(url)).toBe(true); + } + }); + + test('a different port or a real host is never self', () => { + markListening('http://127.0.0.1:4312'); + expect(isSelfOrigin('http://127.0.0.1:4313/x')).toBe(false); + expect(isSelfOrigin('https://api.stripe.com/v1/charges')).toBe(false); + expect(isSelfOrigin('not a url')).toBe(false); + }); + + test('the default port is implied, so origin and bare URL agree', () => { + markListening('https://app.example.com'); + expect(isSelfOrigin('https://app.example.com:443/x')).toBe(true); + expect(isSelfOrigin('https://app.example.com/x')).toBe(true); + expect(isSelfOrigin('http://app.example.com/x')).toBe(false); + }); + + test('release is refcounted and idempotent, so one stop cannot unseal the other server', () => { + const first = markListening('http://127.0.0.1:4314'); + const second = markListening('http://localhost:4314'); + first(); + first(); + expect(isSelfOrigin('http://127.0.0.1:4314/x')).toBe(true); + second(); + expect(isSelfOrigin('http://127.0.0.1:4314/x')).toBe(false); + expect(listeningOrigins()).toEqual([]); + }); + + test('a non-URL origin fails loudly instead of silently never matching', () => { + expect(() => markListening('127.0.0.1:4315')).toThrow(/X_INVARIANT/); + }); +}); diff --git a/packages/core/src/listeners.ts b/packages/core/src/listeners.ts new file mode 100644 index 000000000..ca80dcd2e --- /dev/null +++ b/packages/core/src/listeners.ts @@ -0,0 +1,80 @@ +// Single responsibility: the sockets this process is currently listening on. A request to one of +// them is the process calling itself, not egress — which is how a sealed test network can let a +// booted server reach its own port without an allowlist entry per kernel-assigned port. + +import { assert } from './assert'; + +interface Listener { + readonly origin: string; + count: number; +} + +/** + * Every spelling of "this machine". A wildcard bind is reachable over loopback, so it collapses + * to the same key: a server on `0.0.0.0:3000` answers `http://localhost:3000`. + */ +const LOOPBACK_HOSTS = new Set(['localhost', '127.0.0.1', '0.0.0.0', '[::1]', '[::]']); + +const listeners = new Map(); + +const portOf = (url: URL): string => { + if (url.port !== '') return url.port; + return url.protocol === 'https:' || url.protocol === 'wss:' ? '443' : '80'; +}; + +/** Identity is host+port, not the origin string: the same socket has several valid spellings. */ +const keyOf = (url: URL): string => { + const host = url.hostname.toLowerCase(); + return `${LOOPBACK_HOSTS.has(host) ? 'loopback' : host}:${portOf(url)}`; +}; + +const parseUrl = (url: string): URL | undefined => { + try { + return new URL(url); + } catch { + return undefined; + } +}; + +/** + * Announce a socket this process just opened; call the returned release when it closes. Releasing + * twice is a no-op, so a manual `stop()` and a SIGTERM drain may both call it. Refcounted, so two + * servers on one origin do not un-announce each other. + */ +export function markListening(origin: string): () => void { + const url = parseUrl(origin); + assert( + url !== undefined, + `not a URL: ${origin}`, + 'pass the server origin, e.g. markListening(server.url.origin)', + ); + const key = keyOf(url); + const existing = listeners.get(key); + if (existing === undefined) listeners.set(key, { origin: url.origin, count: 1 }); + else existing.count += 1; + + let released = false; + return () => { + if (released) return; + released = true; + const entry = listeners.get(key); + if (entry === undefined) return; + entry.count -= 1; + if (entry.count <= 0) listeners.delete(key); + }; +} + +/** The origins this process is serving right now, in announce order. */ +export const listeningOrigins = (): readonly string[] => + [...listeners.values()].map((listener) => listener.origin); + +/** True when the URL points at a socket this process opened — same port, any loopback spelling. */ +export function isSelfOrigin(url: string): boolean { + const parsed = parseUrl(url); + return parsed !== undefined && listeners.has(keyOf(parsed)); +} + +/** Test-only: forget every announced socket. */ +export function resetListeners(): void { + listeners.clear(); +} diff --git a/packages/http/CLAUDE.md b/packages/http/CLAUDE.md index 60b0dbfc3..4b8c7c4e4 100644 --- a/packages/http/CLAUDE.md +++ b/packages/http/CLAUDE.md @@ -25,7 +25,8 @@ Owned request lifecycle over `Bun.serve`. Tier 2. `X_UNAUTHENTICATED` is auth's; both are listed in `HTTP_BORROWED_CODES` and filtered out of `registerErrorCodes`. Re-declaring throws `X_ERROR_CODE_DUPLICATE` at import. - Tests must not touch the network — the preload seals `fetch`. Socket tests live in - `e2e/` and run with `ULTIMATE_TEST_ALLOW_NET=1 bun test packages/http/e2e`. + `e2e/` and run with `bun test packages/http/e2e`, sealed: `start()` calls core's + `markListening()`, so the seal treats our own port as self, not egress. Never unseal. ## Files diff --git a/packages/http/e2e/server.e2e.test.ts b/packages/http/e2e/server.e2e.test.ts index a3bedff0a..23778225e 100644 --- a/packages/http/e2e/server.e2e.test.ts +++ b/packages/http/e2e/server.e2e.test.ts @@ -1,7 +1,8 @@ -// Opt-in: binds a real port and makes real requests, so it needs the sealed network -// disabled. Excluded from `bun test` by the root script's --path-ignore-patterns. +// Binds a real port and makes real requests. The network stays sealed — `start()` announces the +// socket to core, so a request back to it is this process calling itself, not egress. Nothing +// excludes this file: it is the `e2e` step of `x verify` and it runs on every push. // -// ULTIMATE_TEST_ALLOW_NET=1 bun test packages/http/e2e +// bun test packages/http/e2e // // What only a socket can prove: Bun's native route table dispatches static paths, the // param fallback still reaches `fetch`, health endpoints answer outside the pipeline, diff --git a/packages/http/src/server.ts b/packages/http/src/server.ts index d35e9db88..a5fd621f1 100644 --- a/packages/http/src/server.ts +++ b/packages/http/src/server.ts @@ -10,6 +10,7 @@ import { healthzPayload, lifecycleState, logger, + markListening, markReady, onShutdown, readyzPayload, @@ -77,6 +78,7 @@ export const createServer = (options: ServerOptions): ServerHandle => { let server: BunServer | undefined; let unregister: (() => void) | undefined; let unregisterClose: (() => void) | undefined; + let stopListening: (() => void) | undefined; /** * Core owns the health state, the in-flight count and the drain deadline so every @@ -144,6 +146,10 @@ export const createServer = (options: ServerOptions): ServerHandle => { fetch: (request, socket) => dispatch(request, socket), }); + // Tell core which socket we opened. A request to it is this process calling itself, + // so the test seal can let it through without an allowlist entry per random port. + stopListening = markListening(server.url.origin); + // 'accept' runs first on SIGTERM: readyz flips to 503 here, while the socket is // still open, so the load balancer stops sending new work before we close it. unregister = onShutdown( @@ -159,6 +165,7 @@ export const createServer = (options: ServerOptions): ServerHandle => { async () => { await server?.stop(true); server = undefined; + stopListening?.(); }, { phase: 'close' }, ); @@ -174,6 +181,9 @@ export const createServer = (options: ServerOptions): ServerHandle => { await drain('manual'); unregister?.(); unregisterClose?.(); + // Idempotent: the close hook already released, unless the drain deadline cut it short. + stopListening?.(); + stopListening = undefined; server = undefined; logger.info(`ultimate ${role} stopped`); }, diff --git a/packages/jobs/src/driver.ts b/packages/jobs/src/driver.ts index 0f9ca0319..dd49dc21f 100644 --- a/packages/jobs/src/driver.ts +++ b/packages/jobs/src/driver.ts @@ -132,3 +132,12 @@ export function setJobDriver(driver: JobDriver): void { export function jobDriver(): JobDriver | undefined { return ambient; } + +/** + * Test/CLI seam: forget the ambient driver. The counterpart to `resetMailDriver()` — a test + * that installs a queue has to be able to put the process back, or every later file in the + * same bun process silently enqueues where it meant to run inline. + */ +export function resetJobDriver(): void { + ambient = undefined; +} diff --git a/packages/jobs/src/index.ts b/packages/jobs/src/index.ts index 89285f62a..3cf576369 100644 --- a/packages/jobs/src/index.ts +++ b/packages/jobs/src/index.ts @@ -18,6 +18,7 @@ export { DEFAULT_QUEUE, DEFAULT_VISIBILITY_TIMEOUT_MS, jobDriver, + resetJobDriver, setJobDriver, } from './driver'; export type { MemoryDriverOptions } from './driver-memory'; diff --git a/packages/mail/src/mail.test.ts b/packages/mail/src/mail.test.ts index 5811a560c..063879ff0 100644 --- a/packages/mail/src/mail.test.ts +++ b/packages/mail/src/mail.test.ts @@ -1,6 +1,7 @@ import { beforeEach, expect, test } from 'bun:test'; import { isUltimateError } from '@ultimat3/core'; import { loadCatalog, registerCatalog } from '@ultimat3/i18n'; +import { resetJobDriver } from '@ultimat3/jobs'; import { t } from '@ultimat3/schema'; import { blocks } from './blocks'; import { registerMailCatalog } from './catalog'; @@ -44,6 +45,10 @@ beforeEach(() => { resetMailDriver(); memory = createMemoryDriver(); setMailDriver(memory); + // `send` enqueues whenever a job driver is ambient, and the driver is process-global. These + // tests assert on the inline path, so they state that precondition instead of inheriting + // whichever driver an earlier file in this bun process happened to leave behind. + resetJobDriver(); }); function codeOf(value: unknown): string { diff --git a/packages/realtime/src/sync-node.ts b/packages/realtime/src/sync-node.ts index df2b9e132..ac9bfb71c 100644 --- a/packages/realtime/src/sync-node.ts +++ b/packages/realtime/src/sync-node.ts @@ -8,6 +8,7 @@ import { type Clock, healthzPayload, logger, + markListening, markReady, onShutdown, readyzPayload, @@ -315,12 +316,21 @@ export function listenSyncNode(node: SyncNode, options: ListenOptions = {}): { s fetch: node.fetch, websocket: node.websocket, }); + // Same rule as @ultimat3/http: every socket the framework opens announces itself, so a request + // back to it is recognisably this process calling itself rather than egress. + const stopListening = markListening(server.url.origin); onShutdown('realtime:sync', async () => { await node.drain(); await node.stop(); server.stop(); + stopListening(); }); - return { stop: () => server.stop() }; + return { + stop: () => { + server.stop(); + stopListening(); + }, + }; } function json(payload: { status: number; body: unknown }): Response { diff --git a/packages/testing/CLAUDE.md b/packages/testing/CLAUDE.md index 2ddcac3fd..2af8e033d 100644 --- a/packages/testing/CLAUDE.md +++ b/packages/testing/CLAUDE.md @@ -2,14 +2,22 @@ Tier 5. May import tiers 0–4. Imported by every package's tests and by generated apps. +Deps: `core` (tier 0), plus `time`, `jobs` and `mail` — imported **dynamically inside the fixture +factories only**, so a test that never destructures `mail` never loads the mail package. + | Rule | Detail | |---|---| | No mocks of the DB | clone a template database; `template-db.ts` is the only DB path | | No wall clock | `frozenClock` / `advanceClock`; `Date.now()` is frozen by the preload | | No unmocked egress | `sealed-network.ts` patches fetch; a miss is `X_TEST_NETWORK_SEALED` | +| Self is not egress | a port core's `markListening()` announced passes through — a socket test never unseals | | No retries | a flake is fixed or deleted the day it flakes; there is no `retry: 3` | -| Test names | always via `testName(type, name)` so `x verify` can filter them | +| Test names | the filename picks the step; `testName(type, name)` on the outer `describe` puts that type on every failure line under it. Never on the inner `test` too — the prefix would print twice | | Injection | `SqlRunner` and `connect` are parameters, so unit tests need no server | +| Fixtures | the preload registers `clock`, `mail`, `runJobs` — an app registers the rest | +| Registry hygiene | the fixture registry is process-global; a test that clears it snapshots with `fixtureSnapshot()` and hands it back in `afterAll` | +| Fixture teardown | a fixture that installs process-global state (the ambient job or mail driver) implements `Symbol.dispose` / `Symbol.asyncDispose` and restores what was there; `fixtureTest` disposes in reverse build order even when the body throws | +| Building one by hand | `createRunJobs()` outside `fixtureTest` is not disposed for you — reset the driver in `afterEach`, or the next file in the process inherits your queue | Commands: `bun test`, `bunx tsc --noEmit -p tsconfig.json`. diff --git a/packages/testing/README.md b/packages/testing/README.md index b05036ddd..7a5263325 100644 --- a/packages/testing/README.md +++ b/packages/testing/README.md @@ -14,6 +14,8 @@ frozen clock. Never let a test reach the network unmocked — it fails by design | `factories.ts` | typed factories from the entity registry, seeded | | `test-types.ts` | the six test types and their helpers | | `matchers.ts` | `toBeUltimateError` `toDenyPolicy` `toEmitSteps` `toMatchOpenApi` `toBeWithinBudget` `toRejectInput` | +| `fixtures.ts` | the registry + `test('…', ({ clock }) => …)` injection | +| `fixture-{clock,mail,jobs}.ts` | the three fixtures the framework owns | | `preload.ts` | the bunfig preload that installs all of the above | ## Install @@ -24,6 +26,46 @@ frozen clock. Never let a test reach the network unmocked — it fails by design preload = ["@ultimat3/testing/preload"] ``` +## Fixtures + +`test` from this package passes a fixture bag as the first argument, and builds only what the +body destructures — a test that never names `runJobs` never starts a queue. + +```ts +import { expect, test } from '@ultimat3/testing'; + +test('the three-day sleep releases the worker', async ({ clock, runJobs }) => { + await runJobs(onboardOrg, { orgId }); + expect(await runJobs.inFlight()).toBe(0); // suspended, not waiting + clock.advance('3d'); + expect(await runJobs.due()).toBe(1); +}); +``` + +| Fixture | Is | Registered by | +|---|---|---| +| `clock` | `now()` · `advance('3d')` · `set(instant)` on the frozen clock | the preload | +| `mail` | `outbox()` · `lastTo(address)` · `failOnce(mail)` over an in-memory transport | the preload | +| `runJobs` | a worker: call it to enqueue+drain, then `drain()` `due()` `inFlight()` `depth()` | the preload | +| anything else | whatever the app registers | the app's `scripts/test-setup.ts` | + +`mail` and `runJobs` install a process-global driver for the length of one test and hand the previous one back afterwards. A fixture that takes over a global does the same: implement `Symbol.dispose` or `Symbol.asyncDispose` on what the factory returns, and `fixtureTest` calls it in reverse build order — including when the test body throws. + +An app adds its own with `defineFixtures` and widens the type by augmenting `Fixtures`: + +```ts +defineFixtures({ seed: () => loadSeed, actorFor: () => actorFor }); + +declare module '@ultimat3/testing' { + interface Fixtures { + readonly seed: (name: string) => SeedHandle; + } +} +``` + +Destructuring a name nobody registered fails with `X_TEST_FIXTURE_UNKNOWN`, which names the set +that *is* registered — never `undefined is not an object` from inside the body. + ## The six test types | Helper | Asserts | `x verify` step | @@ -59,6 +101,11 @@ X_TEST_NETWORK_SEALED fix: mockFetch('https://api.stripe.com/v1/charges', () => new Response('{}')) — or allowHost('api.stripe.com') if it must be real ``` +A server this process booted is exempt: `createServer().start()` announces its socket through +core's `markListening()`, so a test may call its own `handle.url()` on a kernel-assigned port with +the seal fully on. Unsealing (`ULTIMATE_TEST_ALLOW_NET=1`) stays reserved for a deliberate live +integration — never for a socket test. + ## Errors -`X_TEST_NETWORK_SEALED` `X_TEST_DB_UNAVAILABLE` `X_TEST_NONDETERMINISTIC` +`X_TEST_NETWORK_SEALED` `X_TEST_DB_UNAVAILABLE` `X_TEST_NONDETERMINISTIC` `X_TEST_FIXTURE_UNKNOWN` diff --git a/packages/testing/package.json b/packages/testing/package.json index ec8e52a55..ba90fd6c0 100644 --- a/packages/testing/package.json +++ b/packages/testing/package.json @@ -30,6 +30,9 @@ "test": "bun test" }, "dependencies": { - "@ultimat3/core": "0.0.1" + "@ultimat3/core": "0.0.1", + "@ultimat3/jobs": "0.0.1", + "@ultimat3/mail": "0.0.1", + "@ultimat3/time": "0.0.1" } } diff --git a/packages/testing/src/fixture-clock.ts b/packages/testing/src/fixture-clock.ts new file mode 100644 index 000000000..41cf7f326 --- /dev/null +++ b/packages/testing/src/fixture-clock.ts @@ -0,0 +1,28 @@ +// The `clock` fixture: the only legal way for time to pass inside a test. +// +// `clock.advance('3d')` is synchronous — a job test asserts on what is due on the very next +// line — so the duration parser is resolved while the fixture is built, not when it is used. + +import { advanceClock, frozenNow, setFrozenClock } from './determinism'; + +/** `'3d'` | `'30s'` | `1500`. Same vocabulary as a job's `timeout` and a step's `sleep`. */ +export type TestDuration = string | number; + +export interface TestClock { + /** The frozen instant. Never the wall clock. */ + now(): Date; + advance(duration: TestDuration): Date; + set(instant: string | number): Date; +} + +export async function createTestClock(): Promise { + const { toMs } = await import('@ultimat3/time'); + return { + now: frozenNow, + advance: (duration) => advanceClock(toMs(duration)), + set: (instant) => { + setFrozenClock(instant); + return frozenNow(); + }, + }; +} diff --git a/packages/testing/src/fixture-jobs.ts b/packages/testing/src/fixture-jobs.ts new file mode 100644 index 000000000..8defbbb40 --- /dev/null +++ b/packages/testing/src/fixture-jobs.ts @@ -0,0 +1,169 @@ +// The `runJobs` fixture: a whole worker, in-process, driven by the frozen clock. +// +// A job test asserts the guarantees rather than the return value — a step replayed instead of +// re-run, a duplicate enqueue deduped, a `step.sleep('3d')` that parks the run instead of +// holding a connection — so the trace this returns is keyed by step name and counts executions. +// Nothing here polls or sleeps: a job becomes due only because `clock.advance()` said so. + +import { assert, createContext } from '@ultimat3/core'; +import type { + AnyJobHandle, + EnqueueResult, + JobDriver, + JobExecution, + JobHandle, + JobRecord, + StepStatus, +} from '@ultimat3/jobs'; +import { frozenNow } from './determinism'; + +export interface StepTally { + /** Times the step body actually ran. A replay from storage does not count. */ + readonly executions: number; + readonly attempts: number; + readonly status: StepStatus; +} + +/** + * Cumulative for the life of the fixture, which is one test. A retry driven by + * `clock.advance()` is a second drain, and "provision ran once, nudge ran twice" is a claim + * about the whole run — a per-drain trace could not express it. + */ +export interface JobRunTrace { + readonly executions: readonly JobExecution[]; + /** `trace.steps['welcome-email'].executions` — the assertion a job test is written around. */ + readonly steps: Readonly>; +} + +/** `AsyncDisposable`: the fixture installs the ambient job driver and restores it after the test. */ +export interface RunJobs extends AsyncDisposable { + /** Enqueue and drain in one call — the common case. */ + (handle: JobHandle, input: I): Promise; + enqueue(handle: JobHandle, input: I): Promise; + /** Claim and execute everything due at the current instant, until nothing is. */ + drain(): Promise; + /** Live jobs — ready, delayed, running or suspended — optionally for one job only. */ + depth(handle?: AnyJobHandle): Promise; + /** Live jobs claimable right now. `clock.advance()` is what turns delayed into due. */ + due(): Promise; + inFlight(): Promise; +} + +const WORKER_ID = 'test-worker'; +const VISIBILITY_TIMEOUT_MS = 30_000; +const CLAIM_LIMIT = 64; +/** A drain that has not settled in this many rounds is a runaway, not a slow queue. */ +const MAX_ROUNDS = 100; +const LIVE_STATES: ReadonlySet = new Set(['ready', 'delayed', 'running', 'suspended']); + +const tallyOf = (executions: readonly JobExecution[]): Record => { + const steps: Record = {}; + for (const execution of executions) { + const replayed = new Set(execution.replayed); + for (const step of execution.steps) { + const previous = steps[step.name]; + const ran = replayed.has(step.name) ? 0 : 1; + steps[step.name] = { + executions: (previous?.executions ?? 0) + ran, + attempts: step.attempts, + status: step.status, + }; + } + } + return steps; +}; + +export async function createRunJobs(): Promise { + const jobs = await import('@ultimat3/jobs'); + const driver: JobDriver = jobs.createMemoryDriver(); + // Captured before the overwrite: the ambient driver is process-global, so without this the + // next file to call `send()` enqueues into this test's dead queue instead of sending inline. + const previous = jobs.jobDriver(); + jobs.setJobDriver(driver); + const ctx = createContext({ role: 'worker' }); + + const introspect = (): NonNullable => { + const found = driver.introspect; + assert( + found !== undefined, + 'the in-memory job driver lost its introspection surface', + 'runJobs reads queue state through driver.introspect — do not replace the driver inside a test', + ); + return found; + }; + + const live = async (name?: string): Promise => { + const rows = await introspect().list({ limit: 1000, ...(name === undefined ? {} : { name }) }); + return rows.filter((record) => LIVE_STATES.has(record.state)); + }; + + const queues = (): readonly string[] => [ + ...new Set([jobs.DEFAULT_QUEUE, ...jobs.registeredJobs().map((handle) => handle.queue)]), + ]; + + const round = async (): Promise => { + const claimed = await driver.claim({ + queues: queues(), + limit: CLAIM_LIMIT, + visibilityTimeoutMs: VISIBILITY_TIMEOUT_MS, + workerId: WORKER_ID, + }); + const executions: JobExecution[] = []; + for (const job of claimed) { + const handle = jobs.getJob(job.name); + assert( + handle !== undefined, + `queue holds job "${job.name}" but nothing registered it`, + `import the module that declares job("${job.name}") from the test file — the registry is populated by the import, not by the queue`, + ); + executions.push(await jobs.executeJob({ driver, claimed: job, handle, ctx })); + } + return executions; + }; + + /** Every execution this fixture has driven, because the trace is cumulative. */ + const history: JobExecution[] = []; + + const drain = async (): Promise => { + for (let rounds = 0; rounds < MAX_ROUNDS; rounds += 1) { + const batch = await round(); + if (batch.length === 0) return { executions: [...history], steps: tallyOf(history) }; + history.push(...batch); + } + assert( + false, + `runJobs.drain() ran ${MAX_ROUNDS} rounds without the queue settling`, + 'give the failing job a retry delay, or assert with runJobs.due() instead of draining a job that re-enqueues itself', + ); + }; + + const enqueue = async (handle: JobHandle, input: I): Promise => + driver.enqueue({ + name: handle.name, + queue: handle.queue, + input, + idempotencyKey: handle.idempotencyKeyFor(input), + maxAttempts: handle.retry.attempts, + }); + + const enqueueThenDrain = async (handle: JobHandle, input: I): Promise => { + await enqueue(handle, input); + return drain(); + }; + + return Object.assign(enqueueThenDrain, { + enqueue, + drain, + depth: async (handle?: AnyJobHandle) => (await live(handle?.name)).length, + due: async () => + (await live()).filter( + (record) => record.state !== 'running' && record.runAt <= frozenNow().getTime(), + ).length, + inFlight: async () => (await live()).filter((record) => record.state === 'running').length, + [Symbol.asyncDispose]: async (): Promise => { + await driver.close?.(); + if (previous === undefined) jobs.resetJobDriver(); + else jobs.setJobDriver(previous); + }, + }); +} diff --git a/packages/testing/src/fixture-mail.ts b/packages/testing/src/fixture-mail.ts new file mode 100644 index 000000000..910d0de46 --- /dev/null +++ b/packages/testing/src/fixture-mail.ts @@ -0,0 +1,61 @@ +// The `mail` fixture: an in-memory outbox, plus the one failure a mail test actually needs. +// +// `mail.failOnce(nudgeEmail)` is how a job test proves that only the failed step retried. It is +// a transport failure rather than a thrown stub because a stub would bypass rendering, and a +// mail that fails to render is the bug this catches most often. + +import type { MailDriver, MailMessage, SendResult, SentMail } from '@ultimat3/mail'; + +/** A `defineMail()` handle, or its id. Both read naturally at a call site. */ +export type MailRef = string | { readonly id: string }; + +/** `Disposable`: the fixture installs the ambient mail driver and restores it after the test. */ +export interface TestMail extends Disposable { + /** Newest first, so an assertion does not index backwards. */ + outbox(): readonly SentMail[]; + lastTo(address: string): SentMail | undefined; + /** The next send of this mail fails, once. Every later send succeeds. */ + failOnce(mail: MailRef): void; + clear(): void; +} + +const idOf = (mail: MailRef): string => (typeof mail === 'string' ? mail : mail.id); + +export async function createTestMail(): Promise { + const { createMemoryDriver, driverUnavailable, resetMailDriver, setMailDriver, tryMailDriver } = + await import('@ultimat3/mail'); + const memory = createMemoryDriver(); + const failuresLeft = new Map(); + // The ambient driver is process-global; the fixture borrows it for one test and hands it back. + const previous = tryMailDriver(); + + const driver: MailDriver = { + name: 'test', + send(message: MailMessage): Promise { + const left = failuresLeft.get(message.mailId) ?? 0; + if (left === 0) return memory.send(message); + failuresLeft.set(message.mailId, left - 1); + return Promise.reject( + driverUnavailable(`mail.failOnce() failed "${message.mailId}" on purpose`), + ); + }, + }; + setMailDriver(driver); + + return { + outbox: () => memory.outbox(), + lastTo: (address) => memory.lastTo(address), + failOnce: (mail) => { + const id = idOf(mail); + failuresLeft.set(id, (failuresLeft.get(id) ?? 0) + 1); + }, + clear: () => { + memory.clear(); + failuresLeft.clear(); + }, + [Symbol.dispose]: (): void => { + if (previous === undefined) resetMailDriver(); + else setMailDriver(previous); + }, + }; +} diff --git a/packages/testing/src/fixtures.test.ts b/packages/testing/src/fixtures.test.ts index 8804ad0ff..f93507889 100644 --- a/packages/testing/src/fixtures.test.ts +++ b/packages/testing/src/fixtures.test.ts @@ -1,8 +1,24 @@ -import { afterEach, test as bunTest, describe, expect } from 'bun:test'; -import { clearFixtures, defineFixtures, registeredFixtures, requestedFixtures } from './fixtures'; +import { afterAll, beforeEach, test as bunTest, describe, expect } from 'bun:test'; +import { + clearFixtures, + defineFixtures, + fixtureSnapshot, + registeredFixtures, + requestedFixtures, + runWithFixtures, +} from './fixtures'; -afterEach(() => { +// The registry is process-global and the preload filled it. Hand it back, or every file that +// runs after this one loses `clock`, `seed` and the rest — a load-order flake, not a failure. +const preloaded = fixtureSnapshot(); + +beforeEach(() => { + clearFixtures(); +}); + +afterAll(() => { clearFixtures(); + defineFixtures(preloaded); }); describe('requestedFixtures', () => { @@ -28,6 +44,51 @@ describe('requestedFixtures', () => { }); }); +describe('fixtureTest teardown', () => { + // Disposal is what stops one file's ambient driver reaching the next, so it has to hold when a + // disposer itself fails — otherwise one broken fixture re-opens the leak for all of them. + bunTest('a throwing disposer does not strand the fixtures built before it', async () => { + const disposed: string[] = []; + defineFixtures({ + outer: () => ({ [Symbol.dispose]: () => void disposed.push('outer') }), + broken: () => ({ + [Symbol.dispose]: () => { + throw new Error('teardown exploded'); + }, + }), + }); + + // `broken` is built second, so it disposes first — `outer` must still be reached. + const thrown = await runWithFixtures(({ outer, broken }: never) => void [outer, broken]).then( + () => undefined, + (error: unknown) => error, + ); + + expect(disposed).toEqual(['outer']); + expect((thrown as Error).message).toBe('teardown exploded'); + }); + + bunTest('the body’s failure wins over a teardown failure', async () => { + defineFixtures({ + broken: () => ({ + [Symbol.dispose]: () => { + throw new Error('teardown exploded'); + }, + }), + }); + + const thrown = await runWithFixtures(({ broken }: never) => { + void broken; + throw new Error('the assertion that actually broke'); + }).then( + () => undefined, + (error: unknown) => error, + ); + + expect((thrown as Error).message).toBe('the assertion that actually broke'); + }); +}); + describe('defineFixtures', () => { bunTest('merges rather than replaces, so packages can register independently', () => { defineFixtures({ a: () => 1 }); diff --git a/packages/testing/src/fixtures.ts b/packages/testing/src/fixtures.ts index 524f94413..8a32b4027 100644 --- a/packages/testing/src/fixtures.ts +++ b/packages/testing/src/fixtures.ts @@ -10,6 +10,9 @@ import { test as bunTest } from 'bun:test'; import { fixtureUnknown } from './errors'; +import type { TestClock } from './fixture-clock'; +import type { RunJobs } from './fixture-jobs'; +import type { TestMail } from './fixture-mail'; /** Built once per test, on first use. */ export type FixtureFactory = () => T | Promise; @@ -17,18 +20,22 @@ export type FixtureFactory = () => T | Promise; export type FixtureMap = Readonly>; /** - * What a test body receives. Apps widen it by augmenting `Fixtures`: + * What a test body receives. The three the framework owns are declared here and registered by + * the preload; apps widen it by augmenting `Fixtures`: * * ```ts * declare module '@ultimat3/testing' { * interface Fixtures { - * seed: (name: string) => Promise; + * seed: (name: string) => SeedHandle; * } * } * ``` */ -// biome-ignore lint/suspicious/noEmptyInterface: the augmentation target — apps declare into it. -export interface Fixtures {} +export interface Fixtures { + readonly clock: TestClock; + readonly mail: TestMail; + readonly runJobs: RunJobs; +} const registry = new Map(); @@ -44,6 +51,15 @@ export function registeredFixtures(): readonly string[] { return [...registry.keys()].sort(); } +/** + * A copy of the registry. The registry is process-global and bun shares one process across + * files, so a test that needs an empty one snapshots first and hands it back afterwards — + * otherwise every later file silently loses the fixtures the preload registered. + */ +export function fixtureSnapshot(): FixtureMap { + return Object.fromEntries(registry); +} + /** * The names a body destructures, read from its source. * @@ -71,19 +87,75 @@ export function requestedFixtures(body: (...args: never[]) => unknown): readonly export type FixtureBody = (fixtures: Fixtures) => void | Promise; /** - * `test` with fixtures. Only what the body destructures is built, and each is awaited before - * the body runs — so a body reads `seed('dev')` directly instead of awaiting every fixture. + * A fixture that installs process-global state — the ambient job driver, the ambient mail + * driver — implements one of the standard disposal symbols to put it back. Bun shares one + * process across every test file, so a fixture that skips this does not leak within its own + * test: it leaks into every file that runs after it, and the failure surfaces somewhere else. */ -export function fixtureTest(name: string, body: FixtureBody): void { - bunTest(name, async () => { - const wanted = requestedFixtures(body as (...args: never[]) => unknown); - const bag: Record = {}; +type MaybeDisposable = { + readonly [Symbol.asyncDispose]?: () => PromiseLike | void; + readonly [Symbol.dispose]?: () => void; +}; + +const disposerOf = (value: unknown): (() => PromiseLike | void) | undefined => { + if (value === null || (typeof value !== 'object' && typeof value !== 'function')) + return undefined; + const target = value as MaybeDisposable; + const asyncDispose = target[Symbol.asyncDispose]; + if (typeof asyncDispose === 'function') return () => asyncDispose.call(target); + const dispose = target[Symbol.dispose]; + return typeof dispose === 'function' ? () => dispose.call(target) : undefined; +}; + +/** + * Build what the body asked for, run it, dispose in reverse. Split out of `fixtureTest` because + * that one hands its callback to bun and returns nothing — teardown is the part most worth + * testing, and it cannot be observed through a registration. Not in the package's public API: + * `fixtureTest` stays the one way to write a test with fixtures. + */ +export async function runWithFixtures(body: FixtureBody): Promise { + const wanted = requestedFixtures(body as (...args: never[]) => unknown); + // Partial by construction — only what the body destructured is built. Handed over as the + // full `Fixtures` because the keys came from that same body: a key it did not name is a key + // it cannot read, so the missing ones are unobservable. + const bag: Partial & Record = {}; + const built: unknown[] = []; + // Boxed rather than a bare `unknown`, so a body that throws a falsy value still reports. + let failure: { readonly error: unknown } | undefined; + try { for (const key of wanted) { const factory = registry.get(key); // Naming the registered set turns "undefined is not an object" into a fixable message. if (factory === undefined) throw fixtureUnknown(key, registeredFixtures()); - bag[key] = await factory(); + const value = await factory(); + built.push(value); + bag[key] = value; } await body(bag as Fixtures); - }); + } catch (error) { + failure = { error }; + } + + // Every disposer runs even when an earlier one throws: a fixture that cannot clean up must + // not strand the ones built before it. The body's own failure wins, because a teardown error + // that replaced it would hide the assertion that actually broke. + for (const value of built.reverse()) { + try { + await disposerOf(value)?.(); + } catch (error) { + failure ??= { error }; + } + } + if (failure !== undefined) throw failure.error; +} + +/** + * `test` with fixtures. Only what the body destructures is built, and each is awaited before + * the body runs — so a body reads `seed('dev')` directly instead of awaiting every fixture. + * + * Teardown runs in reverse build order whether the body passed or threw: a failing assertion + * must not be the reason the next file inherits a queue. + */ +export function fixtureTest(name: string, body: FixtureBody): void { + bunTest(name, () => runWithFixtures(body)); } diff --git a/packages/testing/src/framework-fixtures.test.ts b/packages/testing/src/framework-fixtures.test.ts new file mode 100644 index 000000000..f6465dee3 --- /dev/null +++ b/packages/testing/src/framework-fixtures.test.ts @@ -0,0 +1,210 @@ +import { afterEach, test as bunTest, describe, expect } from 'bun:test'; +import { assert } from '@ultimat3/core'; +import type { JobDefinition, JobHandle } from '@ultimat3/jobs'; +import { job, jobDriver, resetJobDriver } from '@ultimat3/jobs'; +import { mailDriver, resetMailDriver, tryMailDriver } from '@ultimat3/mail'; +import { frozenNow, setFrozenClock } from './determinism'; +import { createRunJobs } from './fixture-jobs'; +import { createTestMail } from './fixture-mail'; +import { fixtureTest } from './fixtures'; +import { FRAMEWORK_FIXTURE_NAMES, registerFrameworkFixtures } from './framework-fixtures'; +import { testName } from './test-types'; + +// Every global these fixtures touch is process-wide and bun shares one process across files. +// The tests below build fixtures by hand rather than through `fixtureTest`, so nothing disposes +// them for us — the ambient job driver in particular, which turns a later `send()` into an +// enqueue against this file's dead queue. +const START = frozenNow().toISOString(); +afterEach(() => { + setFrozenClock(START); + resetMailDriver(); + resetJobDriver(); +}); + +const DAY_MS = 24 * 60 * 60 * 1_000; + +const passthrough = (): JobDefinition['input'] => ({ + '~standard': { + version: 1, + vendor: 'ultimate-test', + validate: (value: unknown) => ({ value: value as T }), + }, +}); + +const message = (mailId: string) => ({ + mailId, + to: ['ada@acme.example'], + subject: 'mail.welcome.subject', + html: '

hi

', + text: 'hi', + locale: 'en', + tz: 'UTC', +}); + +describe(testName('unit', 'the framework fixture bag'), () => { + bunTest('owns exactly clock, mail and runJobs', () => { + registerFrameworkFixtures(); + expect([...FRAMEWORK_FIXTURE_NAMES]).toEqual(['clock', 'mail', 'runJobs']); + }); + + // The regression the registration exists for: before it, every body destructuring `clock` + // died with X_TEST_FIXTURE_UNKNOWN because nothing in the repo called defineFixtures. + fixtureTest('injects `clock` into a body that destructures it', ({ clock }) => { + expect(clock.now().toISOString()).toBe(frozenNow().toISOString()); + }); + + fixtureTest('clock.advance moves the frozen clock by a duration string', ({ clock }) => { + const before = clock.now().getTime(); + clock.advance('1h'); + expect(clock.now().getTime() - before).toBe(3_600_000); + // The whole point of advancing rather than waiting: `Date.now()` moves with it. + expect(Date.now()).toBe(clock.now().getTime()); + }); +}); + +describe(testName('unit', 'the mail fixture'), () => { + bunTest('failOnce rejects the next send of that mail and only that one', async () => { + const mail = await createTestMail(); + mail.failOnce('welcome'); + + await expect(mailDriver().send(message('welcome'))).rejects.toBeUltimateError( + 'X_MAIL_DRIVER_UNAVAILABLE', + ); + await mailDriver().send(message('welcome')); + await mailDriver().send(message('invite')); + + expect(mail.outbox().map((entry) => entry.message.mailId)).toEqual(['invite', 'welcome']); + }); + + bunTest('failOnce takes a mail definition, not only an id', async () => { + const mail = await createTestMail(); + mail.failOnce({ id: 'invite' }); + + await expect(mailDriver().send(message('invite'))).rejects.toBeUltimateError(); + expect(mail.outbox()).toEqual([]); + }); +}); + +describe(testName('unit', 'the runJobs fixture'), () => { + const flakyJob = (name: string, fails: () => boolean): JobHandle<{ readonly id: string }> => + job<{ readonly id: string }>({ + name, + input: passthrough<{ readonly id: string }>(), + idempotencyKey: (input) => `${name}:${input.id}`, + retry: { attempts: 3, backoff: 'fixed', delay: 1_000, jitter: false }, + run: async ({ step }) => { + await step.run('provision', () => 'provisioned'); + await step.run('nudge', () => { + assert(!fails(), 'nudge failed on purpose', 'nothing — this is a fixture'); + return 'nudged'; + }); + }, + }); + + bunTest('retries only the failed step — the earlier one replays from storage', async () => { + let nudges = 0; + const handle = flakyJob('fixture-retry', () => { + nudges += 1; + return nudges === 1; + }); + const runJobs = await createRunJobs(); + + await runJobs(handle, { id: 'a' }); + setFrozenClock(frozenNow().getTime() + 2_000); + const trace = await runJobs.drain(); + + expect(trace.steps['provision']?.executions).toBe(1); + expect(trace.steps['nudge']?.executions).toBe(2); + expect(await runJobs.depth()).toBe(0); + }); + + bunTest('a duplicate enqueue with a live key returns the same job', async () => { + const handle = flakyJob('fixture-dedupe', () => false); + const runJobs = await createRunJobs(); + + const first = await runJobs.enqueue(handle, { id: 'b' }); + const second = await runJobs.enqueue(handle, { id: 'b' }); + + expect(second.id).toBe(first.id); + expect(second.deduped).toBe(true); + expect(await runJobs.depth(handle)).toBe(1); + }); + + bunTest('a sleeping step parks the run instead of holding a worker', async () => { + const sleeper = job<{ readonly id: string }>({ + name: 'fixture-sleeper', + input: passthrough<{ readonly id: string }>(), + idempotencyKey: (input) => `sleeper:${input.id}`, + retry: { attempts: 1 }, + run: async ({ step }) => { + await step.sleep('3d'); + }, + }); + const runJobs = await createRunJobs(); + + await runJobs(sleeper, { id: 'c' }); + expect(await runJobs.inFlight()).toBe(0); + expect(await runJobs.due()).toBe(0); + + setFrozenClock(frozenNow().getTime() + 3 * DAY_MS); + expect(await runJobs.due()).toBe(1); + }); + + bunTest('each build gets its own queue, so one test cannot see another test’s jobs', async () => { + const handle = flakyJob('fixture-isolated', () => false); + const first = await createRunJobs(); + await first.enqueue(handle, { id: 'e' }); + + const second = await createRunJobs(); + + expect(await first.depth()).toBe(1); + expect(await second.depth()).toBe(0); + }); +}); + +// The regression: `runJobs` used to install the ambient job driver and leave it there. Nothing +// in this file noticed — but `send()` enqueues whenever a queue is ambient, so every mail test +// in a later file asserted on the inline path and got `driver: 'queue'` instead. +describe(testName('unit', 'a fixture that installs process-global state hands it back'), () => { + bunTest('runJobs restores the driver the process had before it', async () => { + resetJobDriver(); + const runJobs = await createRunJobs(); + expect(jobDriver()).toBeDefined(); + + await runJobs[Symbol.asyncDispose](); + + expect(jobDriver()).toBeUndefined(); + }); + + bunTest('and restores an outer driver rather than clearing it', async () => { + const outer = await createRunJobs(); + const outerDriver = jobDriver(); + const inner = await createRunJobs(); + expect(jobDriver()).not.toBe(outerDriver); + + await inner[Symbol.asyncDispose](); + + expect(jobDriver()).toBe(outerDriver); + await outer[Symbol.asyncDispose](); + }); + + bunTest('the mail fixture restores the ambient mail driver too', async () => { + resetMailDriver(); + const mail = await createTestMail(); + expect(tryMailDriver()?.name).toBe('test'); + + mail[Symbol.dispose](); + + expect(tryMailDriver()).toBeUndefined(); + }); + + // Teardown is what the leak fix rides on, so it has to survive the failing test it follows. + fixtureTest('disposal runs even when the body throws', async ({ runJobs }) => { + expect(runJobs).toBeDefined(); + await expect(Promise.reject(new Error('boom'))).rejects.toThrow('boom'); + }); + + bunTest('so the next test starts without the previous one’s queue', () => { + expect(jobDriver()).toBeUndefined(); + }); +}); diff --git a/packages/testing/src/framework-fixtures.ts b/packages/testing/src/framework-fixtures.ts new file mode 100644 index 000000000..3943b3926 --- /dev/null +++ b/packages/testing/src/framework-fixtures.ts @@ -0,0 +1,22 @@ +// Registers the fixtures the FRAMEWORK owns — `clock`, `mail`, `runJobs` — so an app registers +// only what it owns (`seed`, `actorFor`, …). Called by the preload, which is why an app never +// writes `defineFixtures({ clock })` and why two apps cannot disagree about what `clock` means. +// +// Each factory imports its subsystem on demand, for the reason the whole registry is lazy: a +// test that never touches jobs must not pay for the queue, and a `packages/core` test must not +// boot mail. `defineFixtures` merges, so registering twice is idempotent. + +import { createTestClock } from './fixture-clock'; +import { createRunJobs } from './fixture-jobs'; +import { createTestMail } from './fixture-mail'; +import { defineFixtures } from './fixtures'; + +export const FRAMEWORK_FIXTURE_NAMES = ['clock', 'mail', 'runJobs'] as const; + +export function registerFrameworkFixtures(): void { + defineFixtures({ + clock: createTestClock, + mail: createTestMail, + runJobs: createRunJobs, + }); +} diff --git a/packages/testing/src/index.ts b/packages/testing/src/index.ts index 49aff6df5..8f9f6b086 100644 --- a/packages/testing/src/index.ts +++ b/packages/testing/src/index.ts @@ -2,6 +2,7 @@ export type { FixtureBody, FixtureFactory, FixtureMap, Fixtures } from './fixtur export { clearFixtures, defineFixtures, + fixtureSnapshot, fixtureTest, registeredFixtures, requestedFixtures, @@ -40,7 +41,14 @@ export { } from './errors'; export type { EntityLike, EntityRegistry, Factory, FactoryOptions } from './factories'; export { defineFactory, factoriesFor } from './factories'; +export type { TestClock, TestDuration } from './fixture-clock'; +export { createTestClock } from './fixture-clock'; +export type { JobRunTrace, RunJobs, StepTally } from './fixture-jobs'; +export { createRunJobs } from './fixture-jobs'; +export type { MailRef, TestMail } from './fixture-mail'; +export { createTestMail } from './fixture-mail'; export { fixtureTest as test } from './fixtures'; +export { FRAMEWORK_FIXTURE_NAMES, registerFrameworkFixtures } from './framework-fixtures'; export type { AppHandle, AppOptions, BootedApp } from './harness'; export { describeApp, testApp } from './harness'; export type { MatcherResult } from './matchers'; diff --git a/packages/testing/src/preload.ts b/packages/testing/src/preload.ts index 11c9397cc..e7825c872 100644 --- a/packages/testing/src/preload.ts +++ b/packages/testing/src/preload.ts @@ -1,10 +1,12 @@ -// The bunfig preload for apps: frozen clock, seeded RNG, sealed network, custom matchers. Loaded -// once per test process, before any test file — an app never has to remember to call it. +// The bunfig preload for apps: frozen clock, seeded RNG, sealed network, custom matchers, and +// the framework's fixture bag. Loaded once per test process, before any test file — an app never +// has to remember to call it. // // [test] // preload = ["@ultimat3/testing/preload"] import { installDeterminism } from './determinism'; +import { registerFrameworkFixtures } from './framework-fixtures'; import './matchers'; import { sealNetwork } from './sealed-network'; @@ -16,6 +18,8 @@ installDeterminism({ ...(now === undefined ? {} : { now }), }); +registerFrameworkFixtures(); + // Opt-out exists for one case: a test that deliberately exercises a real integration in a job the // team runs on purpose. It is an env var, not an API, so it cannot be set from inside a test file. if (Bun.env['ULTIMATE_TEST_ALLOW_NET'] !== '1') sealNetwork(); diff --git a/packages/testing/src/sealed-network.test.ts b/packages/testing/src/sealed-network.test.ts index ad749456a..37e6563bf 100644 --- a/packages/testing/src/sealed-network.test.ts +++ b/packages/testing/src/sealed-network.test.ts @@ -1,13 +1,23 @@ import { afterEach, describe, expect, test } from 'bun:test'; +import { markListening, resetListeners } from '@ultimat3/core'; import { allowHost, mockJson, requestedUrls, resetNetwork, sealNetwork } from './sealed-network'; +import { testName } from './test-types'; sealNetwork(); afterEach(() => { resetNetwork(); + resetListeners(); }); -describe('unit · sealed network', () => { +/** The seal throws; anything else means the request left this file. Never resolves to a Response. */ +const refusalOf = async (url: string): Promise<{ code?: string } | undefined> => + fetch(url).then( + () => undefined, + (error: unknown) => error as { code?: string }, + ); + +describe(testName('unit', 'sealed network'), () => { test('an unmocked request fails with the URL and the line that fixes it', async () => { try { await fetch('https://api.stripe.com/v1/charges', { method: 'POST' }); @@ -49,6 +59,16 @@ describe('unit · sealed network', () => { } }); + test('a port this process opened is not egress — the seal lets it through', async () => { + const release = markListening('http://127.0.0.1:59321'); + // Nothing is listening, so this must fail — but as a connection error, not as the seal. + expect((await refusalOf('http://localhost:59321/healthz'))?.code).not.toBe( + 'X_TEST_NETWORK_SEALED', + ); + release(); + expect((await refusalOf('http://localhost:59321/healthz'))?.code).toBe('X_TEST_NETWORK_SEALED'); + }); + test('every attempted URL is recorded, in order', async () => { mockJson('https://a.example.com/1', {}); mockJson('https://a.example.com/2', {}); diff --git a/packages/testing/src/sealed-network.ts b/packages/testing/src/sealed-network.ts index 57fa5aa01..554009180 100644 --- a/packages/testing/src/sealed-network.ts +++ b/packages/testing/src/sealed-network.ts @@ -2,6 +2,7 @@ // and the line that fixes it. A test that quietly reaches the internet is a test that fails in CI // for reasons nobody can reproduce — so the default is "nothing gets out". +import { isSelfOrigin } from '@ultimat3/core'; import { NetworkSealedError } from './errors'; export type FetchLike = typeof globalThis.fetch; @@ -50,8 +51,11 @@ export function sealNetwork(): void { if (mock !== undefined) { return mock.handler(input instanceof Request ? input : new Request(url, init)); } + // A server this process booted is not egress — the port is one the kernel just handed us, so + // there is nothing to allowlist ahead of time. Without this, a socket test's only option is to + // unseal the network wholesale, which then hides the real egress it was meant to catch. const host = safeHost(url); - if (host !== undefined && state.allowed.has(host)) { + if (isSelfOrigin(url) || (host !== undefined && state.allowed.has(host))) { const original = state.original; if (original === undefined) throw new TypeError('sealed network lost its original fetch'); return original(input, init); diff --git a/packages/testing/src/test-types.ts b/packages/testing/src/test-types.ts index 7bc94db8f..639377eb6 100644 --- a/packages/testing/src/test-types.ts +++ b/packages/testing/src/test-types.ts @@ -1,6 +1,6 @@ -// The six first-class test types. Each helper prefixes its name with the type, which is what -// `x verify` filters on (`bun test --test-name-pattern "job · "`) — so the six lines in the verify -// output are produced by the tests themselves, not by a directory convention nobody follows. +// The six first-class test types. `x verify` selects a suite by FILENAME — `*.job.test.ts`, and +// so on — so these helpers only prefix the reported name with its type, which is what makes a +// failure line say which of the six steps it belongs to. See packages/cli/src/verify-tests.ts. import { test } from 'bun:test'; diff --git a/packages/testing/tsconfig.json b/packages/testing/tsconfig.json index edba78a9d..e991d81b5 100644 --- a/packages/testing/tsconfig.json +++ b/packages/testing/tsconfig.json @@ -10,6 +10,15 @@ "references": [ { "path": "../core" + }, + { + "path": "../jobs" + }, + { + "path": "../mail" + }, + { + "path": "../time" } ] } diff --git a/packages/ui/src/index.ts b/packages/ui/src/index.ts index 643db508e..a65d7f93a 100644 --- a/packages/ui/src/index.ts +++ b/packages/ui/src/index.ts @@ -1,6 +1,11 @@ // The public surface of @ultimat3/ui. Explicit exports only — this list is the // design system's contract, and anything not named here is internal. +// Ambient declarations are not reachable through imports, so a consuming app's +// program would never load them — every `*.module.scss` import in this package +// would be TS2307 there. The reference pulls the contract along with the entry. +/// + export type { FocusTrap, Politeness, RovingOptions, RovingOrientation } from './a11y'; export { announce, diff --git a/scripts/help.ts b/scripts/help.ts index 1332d2e15..f795a78d9 100644 --- a/scripts/help.ts +++ b/scripts/help.ts @@ -18,7 +18,7 @@ export const SCRIPTS: readonly Entry[] = [ { command: 'bin/check', does: 'the gate — same steps as CI' }, { command: 'bun run scripts/verify.ts', - does: 'typecheck, lint, boundaries, test, filesize, shape', + does: 'the gate — `x verify` at the repo root, all 15 steps', }, { command: 'bun run scripts/boundaries.ts', diff --git a/scripts/lib/steps.ts b/scripts/lib/steps.ts deleted file mode 100644 index 0ca7a704d..000000000 --- a/scripts/lib/steps.ts +++ /dev/null @@ -1,76 +0,0 @@ -// Named steps with their own pass/fail and duration — the same shape `x verify` reports, so the -// framework repo's gate and a generated app's gate print the same thing. - -import type { Finding } from './log'; - -export interface StepOutcome { - readonly ok: boolean; - readonly findings: readonly Finding[]; - readonly output?: string; -} - -export interface Step { - readonly name: string; - readonly summary: string; - run(): Promise; -} - -export interface StepReport { - readonly name: string; - readonly ok: boolean; - readonly durationMs: number; - readonly findings: readonly Finding[]; - readonly output?: string; -} - -export interface RunStepsOptions { - readonly only?: readonly string[]; - readonly skip?: readonly string[]; -} - -/** Never bails early: one run should surface every failure, not the first one. */ -export async function runSteps( - steps: readonly Step[], - options: RunStepsOptions = {}, -): Promise { - const only = options.only ?? []; - const skip = options.skip ?? []; - const chosen = steps.filter( - (step) => (only.length === 0 || only.includes(step.name)) && !skip.includes(step.name), - ); - const reports: StepReport[] = []; - for (const step of chosen) { - const started = performance.now(); - const outcome = await step.run().catch( - (error: unknown): StepOutcome => ({ - ok: false, - findings: [ - { - code: 'X_VERIFY_FAILED', - cause: `step "${step.name}" threw: ${error instanceof Error ? error.message : String(error)}`, - fix: `bun run verify --only ${step.name} --json`, - }, - ], - }), - ); - reports.push({ - name: step.name, - ok: outcome.ok, - durationMs: Math.round(performance.now() - started), - findings: outcome.findings, - ...(outcome.output === undefined ? {} : { output: outcome.output }), - }); - } - return reports; -} - -export const stepLines = (reports: readonly StepReport[], verbose: boolean): readonly string[] => { - const lines: string[] = []; - for (const report of reports) { - lines.push(` ${report.ok ? '✓' : '✗'} ${report.name.padEnd(14)} ${report.durationMs}ms`); - if (report.output !== undefined && (verbose || !report.ok)) { - for (const line of report.output.split('\n').slice(0, 40)) lines.push(` | ${line}`); - } - } - return lines; -}; diff --git a/scripts/test-setup.ts b/scripts/test-setup.ts index e528ac271..6b897d38e 100644 --- a/scripts/test-setup.ts +++ b/scripts/test-setup.ts @@ -1,11 +1,12 @@ // The bunfig preload for the framework repo itself: frozen clock, seeded RNG, sealed network, -// custom matchers. Every package's tests run under the same rules the framework enforces on the -// apps it generates — nondeterminism is a bug here too. +// custom matchers, framework fixtures. Every package's tests run under the same rules the +// framework enforces on the apps it generates — nondeterminism is a bug here too. // // Imported by relative path on purpose: a preload runs before anything else, so it must not depend // on workspace symlinks being installed. Generated apps use `@ultimat3/testing/preload` instead. import { installDeterminism } from '../packages/testing/src/determinism'; +import { registerFrameworkFixtures } from '../packages/testing/src/framework-fixtures'; import '../packages/testing/src/matchers'; import { sealNetwork } from '../packages/testing/src/sealed-network'; @@ -17,5 +18,7 @@ installDeterminism({ ...(now === undefined ? {} : { now }), }); +registerFrameworkFixtures(); + // Opt-out is an env var, not an API, so no test file can quietly unseal the network for itself. if (Bun.env['ULTIMATE_TEST_ALLOW_NET'] !== '1') sealNetwork(); diff --git a/scripts/verify.test.ts b/scripts/verify.test.ts new file mode 100644 index 000000000..f9ce3e343 --- /dev/null +++ b/scripts/verify.test.ts @@ -0,0 +1,37 @@ +import { describe, expect, test } from 'bun:test'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { verifyStepNames } from '@ultimat3/cli'; +import { repoRoot } from './lib/run'; +import { frameworkManifest, HOST_CHECKS, tierBoundaries } from './verify'; + +describe('unit · the repo gate is the CLI gate', () => { + test('the repo adds rules to steps, never steps of its own', () => { + const names: readonly string[] = verifyStepNames(); + for (const step of Object.keys(HOST_CHECKS)) expect(names).toContain(step); + expect(Object.keys(HOST_CHECKS)).toEqual(['boundaries', 'manifest']); + }); + + test('the tier table is enforced through the boundaries step', async () => { + const dir = await mkdtemp(join(tmpdir(), 'ultimate-verify-host-')); + try { + await Bun.write( + join(dir, 'packages/core/src/bad.ts'), + "import { dispatch } from '@ultimat3/cli';\nexport const run = dispatch;\n", + ); + const findings = await tierBoundaries(dir); + expect(findings).toHaveLength(1); + expect(findings[0]?.code).toBe('X_BOUNDARY_VIOLATION'); + expect(findings[0]?.at).toBe('packages/core/src/bad.ts'); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + test('this repo has no tier violations and its manifest still generates', async () => { + const root = repoRoot(); + expect(await tierBoundaries(root)).toEqual([]); + expect(await frameworkManifest(root)).toEqual([]); + }); +}); diff --git a/scripts/verify.ts b/scripts/verify.ts index 6c0d0fe2a..b48dcbe6b 100644 --- a/scripts/verify.ts +++ b/scripts/verify.ts @@ -1,154 +1,49 @@ #!/usr/bin/env bun -// The gate for the framework repo itself — the same named steps `x verify` runs for an app, so a -// contributor and a user see the same output. Green means shippable. +// The gate for the framework repo itself: `x verify`, run at the repo root. The step list, the +// runner, the report and the exit code all come from @ultimat3/cli — a contributor and a user see +// the same steps because there is only one list. This file adds the two rules a package monorepo +// enforces that the CLI cannot know on its own: the tier table, and its generated manifest. // -// bun run scripts/verify.ts [--only lint,boundaries] [--skip test] [--json] [--verbose] +// bun run scripts/verify.ts [--json] [--verbose] -import { existsSync } from 'node:fs'; -import { join } from 'node:path'; +import type { HostCheck, VerifyStepName } from '@ultimat3/cli'; +import { exec, exitCodeFor, render, runVerify, VERIFY_STEPS } from '@ultimat3/cli'; import { checkBoundaries, collectSourceFiles, findingFor } from './boundaries'; -import { flagBool, flagList, parseScriptArgs } from './lib/args'; -import type { Finding } from './lib/log'; -import { report } from './lib/log'; -import { repoRoot, run } from './lib/run'; -import type { Step, StepOutcome } from './lib/steps'; -import { runSteps, stepLines } from './lib/steps'; -import { listWorkspaces } from './lib/workspaces'; +import { flagBool, parseScriptArgs } from './lib/args'; +import { repoRoot } from './lib/run'; +import { buildManifest, DEFAULT_OUT } from './manifest'; -const HARD_LINE_CEILING = 500; +/** The tier table, enforced: a package may import only from a strictly lower tier. */ +export const tierBoundaries: HostCheck = async (root) => + checkBoundaries(await collectSourceFiles(root)).map(findingFor); -const fromRun = async ( - command: readonly string[], - root: string, - finding: Finding, -): Promise => { - const result = await run(command, { cwd: root }); - return result.ok - ? { ok: true, findings: [], output: result.output } - : { ok: false, findings: [finding], output: result.output }; -}; - -/** Files are the unit of review: one file, one job, hard ceiling ~500 lines. */ -async function checkFileSizes(root: string): Promise { - const findings: Finding[] = []; - for (const pattern of ['packages/*/src/**/*.ts', 'scripts/**/*.ts']) { - for await (const path of new Bun.Glob(pattern).scan({ cwd: root, absolute: false })) { - const lines = (await Bun.file(join(root, path)).text()).split('\n').length; - if (lines > HARD_LINE_CEILING) { - findings.push({ - code: 'X_FILE_TOO_LONG', - cause: `${path} is ${lines} lines, over the ${HARD_LINE_CEILING} line ceiling`, - fix: 'split it: one file, one responsibility', - at: path, - }); - } - } - } - return { ok: findings.length === 0, findings }; -} - -/** Every framework package ships the same seven files; a missing one is a build error. */ -async function checkPackageShape(root: string): Promise { - const findings: Finding[] = []; - for (const workspace of await listWorkspaces(root)) { - for (const required of ['README.md', 'CLAUDE.md', 'tsconfig.json', 'src/index.ts']) { - if (existsSync(join(workspace.path, required))) continue; - findings.push({ - code: 'X_PACKAGE_SHAPE', - cause: `${workspace.name} has no ${required}`, - fix: `bun run scripts/new-package.ts ${workspace.dir} --tier ${workspace.tier} --only ${required}`, - at: `packages/${workspace.dir}/${required}`, - }); - } +/** The framework's own manifest is generated from the packages; it must still generate. */ +export const frameworkManifest: HostCheck = async (root) => { + try { + await buildManifest(root); + return []; + } catch (error) { + return [ + { + code: 'X_MANIFEST_STALE', + cause: `the framework manifest could not be generated: ${error instanceof Error ? error.message : String(error)}`, + fix: 'bun run manifest', + docs: 'https://ultimate.dev/errors/X_MANIFEST_STALE', + at: DEFAULT_OUT, + }, + ]; } - return { ok: findings.length === 0, findings }; -} +}; -function stepsFor(root: string): readonly Step[] { - return [ - { - name: 'typecheck', - summary: 'tsc -b across every package', - run: () => - fromRun(['bunx', 'tsc', '-b', '--pretty', 'false'], root, { - code: 'X_TYPECHECK_FAILED', - cause: 'the workspace does not typecheck', - fix: 'bunx tsc -b --pretty false', - }), - }, - { - name: 'lint', - summary: 'biome: no any, no default exports, formatting', - run: () => - fromRun(['bunx', 'biome', 'check', '.'], root, { - code: 'X_LINT_FAILED', - cause: 'biome reported problems', - fix: 'bunx biome check --write .', - }), - }, - { - name: 'boundaries', - summary: 'the tier table, enforced', - run: async () => { - const violations = checkBoundaries(await collectSourceFiles(root)); - return { ok: violations.length === 0, findings: violations.map(findingFor) }; - }, - }, - { - name: 'test', - summary: 'bun test across every package', - // `bun run test`, not bare `bun test`, so this gate and CI run the same command. - // The script's ignore patterns drop two kinds of file: the `e2e/` suites, which - // bind a real socket and are opt-in behind ULTIMATE_TEST_ALLOW_NET=1; and the - // dummy app's contract/live/job/eval suites, which request fixtures nothing in - // the repo registers — 13 tests that have never passed anywhere. See #9. - run: () => - fromRun(['bun', 'run', 'test'], root, { - code: 'X_TEST_FAILED', - cause: 'one or more tests failed', - fix: 'bun run test', - }), - }, - { name: 'filesize', summary: 'one file, one job', run: () => checkFileSizes(root) }, - { - name: 'package-shape', - summary: 'every package ships the same files', - run: () => checkPackageShape(root), - }, - { - name: 'manifest', - summary: 'the framework manifest regenerates', - run: () => - fromRun(['bun', 'run', 'scripts/manifest.ts', '--json'], root, { - code: 'X_MANIFEST_STALE', - cause: 'the framework manifest could not be generated', - fix: 'bun run scripts/manifest.ts', - }), - }, - ]; -} +export const HOST_CHECKS: Partial> = { + boundaries: tierBoundaries, + manifest: frameworkManifest, +}; if (import.meta.main) { const args = parseScriptArgs(Bun.argv.slice(2)); const root = repoRoot(); - const reports = await runSteps(stepsFor(root), { - only: flagList(args, 'only'), - skip: flagList(args, 'skip'), - }); - const failed = reports.filter((step) => !step.ok); - const totalMs = reports.reduce((sum, step) => sum + step.durationMs, 0); - report( - { - ok: failed.length === 0, - script: 'verify', - summary: - failed.length === 0 - ? `all ${reports.length} steps passed in ${totalMs}ms` - : `${failed.length} of ${reports.length} steps failed`, - findings: failed.flatMap((step) => step.findings), - lines: stepLines(reports, flagBool(args, 'verbose')), - data: { steps: reports.map(({ output: _output, ...rest }) => rest), durationMs: totalMs }, - }, - args.json, - ); + const result = await runVerify(VERIFY_STEPS, { root, runner: exec, hostChecks: HOST_CHECKS }); + process.stdout.write(`${render(result, args.json, flagBool(args, 'verbose'))}\n`); + process.exit(exitCodeFor(result)); } diff --git a/site/build.ts b/site/build.ts index 01b1c6e32..cca35ca85 100644 --- a/site/build.ts +++ b/site/build.ts @@ -5,549 +5,15 @@ // because a marketing site that claims "no dependency runtime" has to mean it. import { mkdirSync, rmSync } from 'node:fs'; - -const ORIGIN = 'https://ultimate.developerz.ai'; -const ROOT = new URL('.', import.meta.url).pathname.replace(/\/$/, ''); -const REPO_ROOT = ROOT.replace(/\/site$/, ''); -const DIST = `${ROOT}/dist`; -const STYLE_ORDER = ['tokens', 'base', 'layout', 'components', 'syntax'] as const; -const PAGE_ORDER = [ - 'index', - 'quickstart', - 'primitives', - 'realtime', - 'jobs', - 'rendering-seo', - 'pwa-offline', - 'ai-first', - 'deploy', - 'roadmap', - 'faq', - 'changelog', -] as const; - -interface Page { - readonly slug: string; - readonly url: string; - readonly file: string; - readonly meta: Record; - readonly body: string; -} - -// ─────────────────────────────────────────────────────────────── text helpers - -const escapeHtml = (s: string): string => - s - .replaceAll('&', '&') - .replaceAll('<', '<') - .replaceAll('>', '>') - .replaceAll('"', '"'); - -const slugify = (s: string): string => - s - .toLowerCase() - .replace(/<[^>]+>/g, '') - .replace(/[^\w\s-]/g, '') - .trim() - .replace(/\s+/g, '-'); - -const fill = (template: string, vars: Record): string => - template.replace(/\{\{(\w+)\}\}/g, (_, key: string) => vars[key] ?? ''); - -function frontmatter(src: string): { meta: Record; body: string } { - if (!src.startsWith('---\n')) return { meta: {}, body: src }; - const end = src.indexOf('\n---', 4); - const meta: Record = {}; - for (const line of src.slice(4, end).split('\n')) { - const at = line.indexOf(':'); - if (at < 1) continue; - meta[line.slice(0, at).trim()] = line - .slice(at + 1) - .trim() - .replace(/^"(.*)"$/, '$1'); - } - return { meta, body: src.slice(end + 4).replace(/^\n+/, '') }; -} - -// ──────────────────────────────────────────────────────── syntax highlighting - -const KEYWORDS = - 'as|async|await|break|case|catch|class|const|continue|declare|default|delete|do|else|' + - 'enum|export|extends|false|finally|for|from|function|get|if|implements|import|in|' + - 'instanceof|interface|let|new|null|of|readonly|return|satisfies|set|static|super|' + - 'switch|this|throw|true|try|type|typeof|undefined|var|void|while|yield'; - -const CODE_RE = new RegExp( - [ - '(\\/\\/[^\\n]*|\\/\\*[\\s\\S]*?\\*\\/|#[^\\n]*)', // 1 comment - '(\'(?:\\\\.|[^\'\\\\])*\'|"(?:\\\\.|[^"\\\\])*"|`(?:\\\\.|[^`\\\\])*`)', // 2 string - '\\b(\\d[\\d_.]*[a-z]{0,2})\\b', // 3 number - `\\b(${KEYWORDS})\\b`, // 4 keyword - '\\b([A-Z][A-Za-z0-9_]*)\\b', // 5 type - '\\b([a-zA-Z_$][\\w$]*)(?=\\s*\\()', // 6 call - ].join('|'), - 'g', -); - -const CLASSES = ['tok-comment', 'tok-string', 'tok-number', 'tok-keyword', 'tok-type', 'tok-fn']; - -/** Token classes only — no AST, no grammar file. Good enough for the shapes we ship. */ -function highlightCode(source: string): string { - let out = ''; - let last = 0; - for (const match of source.matchAll(CODE_RE)) { - const at = match.index ?? 0; - out += escapeHtml(source.slice(last, at)); - const group = CLASSES.findIndex((_, i) => match[i + 1] !== undefined); - out += `${escapeHtml(match[0])}`; - last = at + match[0].length; - } - return out + escapeHtml(source.slice(last)); -} - -/** Terminal transcripts: prompt, command, pass/fail marks, and the 3-line error shape. */ -function highlightShell(source: string): string { - return source - .split('\n') - .map((line) => { - const prompt = /^(\s*)\$ (.*)$/.exec(line); - if (prompt !== null) { - return `${prompt[1]}$ ${escapeHtml(prompt[2] ?? '')}`; - } - const mark = /^(\s*)([✓✗])(.*)$/.exec(line); - if (mark !== null) { - const cls = mark[2] === '✓' ? 'tok-pass' : 'tok-fail'; - return `${mark[1]}${mark[2]}${escapeHtml(mark[3] ?? '')}`; - } - const code = /^(\s*)(X_[A-Z0-9_]+)(:.*)$/.exec(line); - if (code !== null) { - return `${code[1]}${code[2]}${escapeHtml(code[3] ?? '')}`; - } - const label = /^(\s+)(cause|fix|docs):(\s*)(.*)$/.exec(line); - if (label !== null) { - return `${label[1]}${label[2]}:${label[3]}${escapeHtml(label[4] ?? '')}`; - } - const diff = /^([+-])(.*)$/.exec(line); - if (diff !== null) { - const cls = diff[1] === '+' ? 'tok-added' : 'tok-removed'; - return `${escapeHtml(line)}`; - } - return escapeHtml(line); - }) - .join('\n'); -} - -const SHELL_LANGS = new Set(['bash', 'sh', 'shell', 'console', 'text', 'diff', 'yaml', '']); - -function renderCode(lang: string, title: string | undefined, source: string): string { - const body = SHELL_LANGS.has(lang) ? highlightShell(source) : highlightCode(source); - const cls = SHELL_LANGS.has(lang) && lang !== 'yaml' ? ' class="terminal"' : ''; - const pre = `${body}`; - if (title === undefined) return pre; - return `
${escapeHtml(title)}${escapeHtml(lang)}
${pre}
`; -} - -// ─────────────────────────────────────────────────────────────────── markdown - -function inline(src: string): string { - return src - .split(/(`[^`]+`)/g) - .map((part) => { - if (part.length > 1 && part.startsWith('`') && part.endsWith('`')) { - return `${escapeHtml(part.slice(1, -1))}`; - } - return escapeHtml(part) - .replace(/\[([^\]]+)\]\(([^)\s]+)\)/g, '$1') - .replace(/\*\*([^*]+)\*\*/g, '$1') - .replace(/(^|[\s(])\*([^*\n]+)\*/g, '$1$2'); - }) - .join(''); -} - -const cells = (row: string): string[] => - row - .replace(/^\||\|$/g, '') - .split('|') - .map((c) => c.trim()); - -interface Rendered { - readonly html: string; - readonly headings: readonly { id: string; text: string }[]; -} - -/** Block-level markdown: headings, lists, tables, fences, callouts, raw HTML. */ -function markdown(src: string): Rendered { - const lines = src.split('\n'); - const headings: { id: string; text: string }[] = []; - const out: string[] = []; - let i = 0; - - const paragraph = (buffer: string[]): void => { - if (buffer.length > 0) out.push(`

${inline(buffer.join(' '))}

`); - buffer.length = 0; - }; - - while (i < lines.length) { - const line = lines[i] ?? ''; - - if (line.trim() === '') { - i += 1; - continue; - } - - // fenced code - const fence = /^```(\S*)(?:\s+title="([^"]*)")?\s*$/.exec(line); - if (fence !== null) { - const buffer: string[] = []; - i += 1; - while (i < lines.length && !(lines[i] ?? '').startsWith('```')) { - buffer.push(lines[i] ?? ''); - i += 1; - } - i += 1; - out.push(renderCode(fence[1] ?? '', fence[2], buffer.join('\n'))); - continue; - } - - // ::: callout … ::: - const callout = /^:::\s*(ok|warn|info)(?:\s+(.*))?$/.exec(line); - if (callout !== null) { - const buffer: string[] = []; - i += 1; - while (i < lines.length && (lines[i] ?? '').trim() !== ':::') { - buffer.push(lines[i] ?? ''); - i += 1; - } - i += 1; - const label = callout[2] ?? callout[1] ?? 'note'; - const inner = markdown(buffer.join('\n')).html; - out.push( - ``, - ); - continue; - } - - // raw HTML block — passed through untouched, so the pitch can use the grid components - if (line.startsWith('<')) { - const buffer: string[] = []; - while (i < lines.length && (lines[i] ?? '').trim() !== '') { - buffer.push(lines[i] ?? ''); - i += 1; - } - out.push(buffer.join('\n')); - continue; - } - - // heading - const heading = /^(#{2,4})\s+(.*)$/.exec(line); - if (heading !== null) { - const level = (heading[1] ?? '##').length; - const text = heading[2] ?? ''; - const id = slugify(text); - if (level === 2) headings.push({ id, text }); - out.push( - `${inline(text)}` + - `#`, - ); - i += 1; - continue; - } - - // table - if (line.startsWith('|') && /^\|[\s:|-]+\|$/.test(lines[i + 1] ?? '')) { - const head = cells(line); - i += 2; - const rows: string[][] = []; - while (i < lines.length && (lines[i] ?? '').startsWith('|')) { - rows.push(cells(lines[i] ?? '')); - i += 1; - } - const thead = head.map((c) => `${inline(c)}`).join(''); - const tbody = rows - .map((row) => `${row.map((c) => `${inline(c)}`).join('')}`) - .join(''); - out.push( - `
${thead}${tbody}
`, - ); - continue; - } - - // lists - const bullet = /^[-*]\s+(.*)$/.exec(line); - const ordered = /^\d+\.\s+(.*)$/.exec(line); - if (bullet !== null || ordered !== null) { - const tag = bullet !== null ? 'ul' : 'ol'; - const items: string[] = []; - while (i < lines.length) { - const current = lines[i] ?? ''; - const item = /^(?:[-*]|\d+\.)\s+(.*)$/.exec(current); - if (item !== null) { - items.push(item[1] ?? ''); - } else if (/^\s+\S/.test(current) && items.length > 0) { - items[items.length - 1] += ` ${current.trim()}`; - } else { - break; - } - i += 1; - } - out.push(`<${tag}>${items.map((t) => `
  • ${inline(t)}
  • `).join('')}`); - continue; - } - - // blockquote - if (line.startsWith('> ')) { - const buffer: string[] = []; - while (i < lines.length && (lines[i] ?? '').startsWith('>')) { - buffer.push((lines[i] ?? '').replace(/^>\s?/, '')); - i += 1; - } - out.push(`
    ${markdown(buffer.join('\n')).html}
    `); - continue; - } - - if (/^(---|\*\*\*)\s*$/.test(line)) { - out.push('
    '); - i += 1; - continue; - } - - // paragraph - const buffer: string[] = []; - while (i < lines.length) { - const current = lines[i] ?? ''; - if ( - current.trim() === '' || - current.startsWith('#') || - current.startsWith('|') || - current.startsWith('```') || - current.startsWith(':::') || - current.startsWith('<') || - current.startsWith('> ') || - /^(?:[-*]|\d+\.)\s/.test(current) - ) { - break; - } - buffer.push(current.trim()); - i += 1; - } - paragraph(buffer); - } - - return { html: out.join('\n'), headings }; -} - -// ────────────────────────────────────────────────────────────────── assembly - -async function readText(path: string): Promise { - return await Bun.file(path).text(); -} - -/** SCSS here is nesting + custom properties only, so stripping `//` comments is a compile. */ -async function compileStyles(): Promise { - const parts: string[] = []; - for (const name of STYLE_ORDER) { - const src = await readText(`${ROOT}/styles/${name}.scss`); - parts.push( - src - .split('\n') - .filter((line) => !/^\s*\/\//.test(line)) - .join('\n'), - ); - } - return parts - .join('\n') - .replace(/\/\*[\s\S]*?\*\//g, '') - .replace(/\n{2,}/g, '\n') - .replace(/[ \t]+/g, ' ') - .replace(/\s*([{;:,>])\s*/g, '$1') - .replace(/;}/g, '}') - .trim(); -} - -async function compileScript(): Promise { - const src = await readText(`${ROOT}/scripts/theme.js`); - return src - .split('\n') - .filter((line) => !/^\s*\/\//.test(line)) - .join('\n') - .replace(/\/\*[\s\S]*?\*\//g, '') - .replace(/\s+/g, ' ') - .trim(); -} - -function seoCheck(pages: readonly Page[]): void { - const titles = new Map(); - const descriptions = new Map(); - for (const page of pages) { - const title = page.meta.title ?? ''; - const description = page.meta.description ?? ''; - if (title === '') throw new Error(`X_SEO_NO_TITLE: ${page.file} has no title in frontmatter`); - if (description === '') { - throw new Error(`X_SEO_NO_DESCRIPTION: ${page.file} has no description in frontmatter`); - } - if (description.length < 50 || description.length > 160) { - throw new Error( - `X_SEO_NO_DESCRIPTION: ${page.file} description is ${description.length} chars, needs 50-160`, - ); - } - const dupeTitle = titles.get(title); - if (dupeTitle !== undefined) { - throw new Error(`X_SEO_DUPLICATE_TITLE: ${page.file} repeats the title of ${dupeTitle}`); - } - const dupeDescription = descriptions.get(description); - if (dupeDescription !== undefined) { - throw new Error( - `X_SEO_DUPLICATE_DESCRIPTION: ${page.file} repeats the description of ${dupeDescription}`, - ); - } - titles.set(title, page.file); - descriptions.set(description, page.file); - } -} - -function navHtml(pages: readonly Page[], current: Page): string { - return ( - pages - // `menu: true` opts a page into the header; every page is still reachable from the - // footer and the pager, so the header never has to grow a scroll region on desktop. - .filter((page) => page.meta.menu === 'true') - .map((page) => { - const active = page.slug === current.slug ? ' aria-current="page"' : ''; - return `
  • ${escapeHtml(page.meta.nav ?? '')}
  • `; - }) - .join('\n') - ); -} - -function tocHtml(headings: readonly { id: string; text: string }[]): string { - if (headings.length === 0) return ''; - const items = headings - .map((h) => `
  • ${inline(h.text)}
  • `) - .join('\n'); - return `
      \n${items}\n
    `; -} - -function pagerHtml(pages: readonly Page[], index: number): string { - const previous = pages[index - 1]; - const next = pages[index + 1]; - if (previous === undefined && next === undefined) return ''; - const left = - previous === undefined - ? '' - : `Previous${escapeHtml(previous.meta.nav ?? previous.slug)}`; - const right = - next === undefined - ? '' - : `Next${escapeHtml(next.meta.nav ?? next.slug)}`; - return ` `; -} - -function jsonLd(page: Page, isHome: boolean): string { - const graph = isHome - ? { - '@context': 'https://schema.org', - '@type': 'SoftwareApplication', - name: 'Ultimate', - applicationCategory: 'DeveloperApplication', - applicationSubCategory: 'Web framework', - operatingSystem: 'Linux, macOS', - description: page.meta.description, - url: `${ORIGIN}/`, - softwareVersion: '0.0.1', - license: 'https://opensource.org/licenses/MIT', - codeRepository: 'https://github.com/developerz-ai/ultimate', - runtimePlatform: 'Bun >= 1.3', - offers: { '@type': 'Offer', price: '0', priceCurrency: 'USD' }, - author: { '@type': 'Organization', name: 'developerz-ai' }, - } - : { - '@context': 'https://schema.org', - '@type': 'TechArticle', - headline: page.meta.title, - description: page.meta.description, - url: `${ORIGIN}${page.url}`, - dateModified: page.meta.updated, - inLanguage: 'en', - isPartOf: { '@type': 'WebSite', name: 'Ultimate', url: `${ORIGIN}/` }, - author: { '@type': 'Organization', name: 'developerz-ai' }, - proficiencyLevel: 'Expert', - }; - const crumbs = { - '@context': 'https://schema.org', - '@type': 'BreadcrumbList', - itemListElement: [ - { '@type': 'ListItem', position: 1, name: 'Ultimate', item: `${ORIGIN}/` }, - ...(isHome - ? [] - : [ - { - '@type': 'ListItem', - position: 2, - name: page.meta.title, - item: `${ORIGIN}${page.url}`, - }, - ]), - ], - }; - return [graph, crumbs] - .map( - (data) => - // `<` is escaped so no value can close the script element early. - ` `, - ) - .join('\n'); -} - -function feedXml(changelog: Page): string { - // Sections are split by hand rather than by one regex: `$` under /m would end the - // capture at the first line break and every description would ship empty. - const items = changelog.body - .split(/^## /m) - .map((section) => /^(.+?) — (\d{4}-\d{2}-\d{2})\n([\s\S]*)$/.exec(section)) - .filter((match): match is RegExpExecArray => match !== null) - .slice(0, 20) - .map((match) => { - const [, version = '', date = '', body = ''] = match; - const id = slugify(`${version} ${date}`); - const summary = body - .split('\n') - .filter((line) => line.startsWith('- ')) - .map((line) => line.slice(2).replace(/[`*[\]]/g, '')) - .join(' · '); - return ` - ${escapeHtml(version)} - ${ORIGIN}/changelog/#${id} - ultimate-${id} - ${new Date(`${date}T00:00:00Z`).toUTCString()} - ${escapeHtml(summary)} - `; - }) - .join('\n'); - return ` - - - Ultimate changelog - ${ORIGIN}/changelog/ - - Releases and milestone notes for the Ultimate framework. - en - ${new Date().toUTCString()} -${items} - - -`; -} - -async function copyDir(from: string, to: string): Promise { - let count = 0; - for await (const entry of new Bun.Glob('**/*').scan({ cwd: from, onlyFiles: true })) { - await Bun.write(`${to}/${entry}`, Bun.file(`${from}/${entry}`)); - count += 1; - } - return count; -} - -// ────────────────────────────────────────────────────────────────────── build +import { compileScript, compileStyles, copyDir, readText } from './lib/assets'; +import type { Page } from './lib/config'; +import { DIST, ORIGIN, PAGE_ORDER, REPO_ROOT, ROOT } from './lib/config'; +import { feedXml } from './lib/feed'; +import { renderCode } from './lib/highlight'; +import { jsonLd, navHtml, pagerHtml, tocHtml } from './lib/html'; +import { inline, markdown } from './lib/markdown'; +import { seoCheck } from './lib/seo'; +import { escapeHtml, fill, frontmatter } from './lib/text'; async function build(): Promise { const started = Bun.nanoseconds(); diff --git a/site/lib/assets.ts b/site/lib/assets.ts new file mode 100644 index 000000000..07254c34f --- /dev/null +++ b/site/lib/assets.ts @@ -0,0 +1,50 @@ +// Everything the build reads off disk and reshapes on the way out: source text, the inlined SCSS +// bundle, the inline theme script, and the static asset copy. + +import { ROOT, STYLE_ORDER } from './config'; + +export async function readText(path: string): Promise { + return await Bun.file(path).text(); +} + +/** SCSS here is nesting + custom properties only, so stripping `//` comments is a compile. */ +export async function compileStyles(): Promise { + const parts: string[] = []; + for (const name of STYLE_ORDER) { + const src = await readText(`${ROOT}/styles/${name}.scss`); + parts.push( + src + .split('\n') + .filter((line) => !/^\s*\/\//.test(line)) + .join('\n'), + ); + } + return parts + .join('\n') + .replace(/\/\*[\s\S]*?\*\//g, '') + .replace(/\n{2,}/g, '\n') + .replace(/[ \t]+/g, ' ') + .replace(/\s*([{;:,>])\s*/g, '$1') + .replace(/;}/g, '}') + .trim(); +} + +export async function compileScript(): Promise { + const src = await readText(`${ROOT}/scripts/theme.js`); + return src + .split('\n') + .filter((line) => !/^\s*\/\//.test(line)) + .join('\n') + .replace(/\/\*[\s\S]*?\*\//g, '') + .replace(/\s+/g, ' ') + .trim(); +} + +export async function copyDir(from: string, to: string): Promise { + let count = 0; + for await (const entry of new Bun.Glob('**/*').scan({ cwd: from, onlyFiles: true })) { + await Bun.write(`${to}/${entry}`, Bun.file(`${from}/${entry}`)); + count += 1; + } + return count; +} diff --git a/site/lib/config.ts b/site/lib/config.ts new file mode 100644 index 000000000..e548823ec --- /dev/null +++ b/site/lib/config.ts @@ -0,0 +1,31 @@ +// The constants every stage of the site build shares: origin, source and output roots, and the +// fixed stylesheet and page order. Defined once here so no two stages can disagree. + +export const ORIGIN = 'https://ultimate.developerz.ai'; +// `..` because this module sits in site/lib/ — ROOT must still resolve to the site directory. +export const ROOT = new URL('..', import.meta.url).pathname.replace(/\/$/, ''); +export const REPO_ROOT = ROOT.replace(/\/site$/, ''); +export const DIST = `${ROOT}/dist`; +export const STYLE_ORDER = ['tokens', 'base', 'layout', 'components', 'syntax'] as const; +export const PAGE_ORDER = [ + 'index', + 'quickstart', + 'primitives', + 'realtime', + 'jobs', + 'rendering-seo', + 'pwa-offline', + 'ai-first', + 'deploy', + 'roadmap', + 'faq', + 'changelog', +] as const; + +export interface Page { + readonly slug: string; + readonly url: string; + readonly file: string; + readonly meta: Record; + readonly body: string; +} diff --git a/site/lib/feed.ts b/site/lib/feed.ts new file mode 100644 index 000000000..7029715ac --- /dev/null +++ b/site/lib/feed.ts @@ -0,0 +1,46 @@ +// The RSS feed, derived from the changelog page's own markdown so a release is written once and +// published twice. + +import type { Page } from './config'; +import { ORIGIN } from './config'; +import { escapeHtml, slugify } from './text'; + +export function feedXml(changelog: Page): string { + // Sections are split by hand rather than by one regex: `$` under /m would end the + // capture at the first line break and every description would ship empty. + const items = changelog.body + .split(/^## /m) + .map((section) => /^(.+?) — (\d{4}-\d{2}-\d{2})\n([\s\S]*)$/.exec(section)) + .filter((match): match is RegExpExecArray => match !== null) + .slice(0, 20) + .map((match) => { + const [, version = '', date = '', body = ''] = match; + const id = slugify(`${version} ${date}`); + const summary = body + .split('\n') + .filter((line) => line.startsWith('- ')) + .map((line) => line.slice(2).replace(/[`*[\]]/g, '')) + .join(' · '); + return ` + ${escapeHtml(version)} + ${ORIGIN}/changelog/#${id} + ultimate-${id} + ${new Date(`${date}T00:00:00Z`).toUTCString()} + ${escapeHtml(summary)} + `; + }) + .join('\n'); + return ` + + + Ultimate changelog + ${ORIGIN}/changelog/ + + Releases and milestone notes for the Ultimate framework. + en + ${new Date().toUTCString()} +${items} + + +`; +} diff --git a/site/lib/highlight.ts b/site/lib/highlight.ts new file mode 100644 index 000000000..76110c919 --- /dev/null +++ b/site/lib/highlight.ts @@ -0,0 +1,80 @@ +// Syntax highlighting for fenced code blocks: a token pass for source, a line pass for terminal +// transcripts, and the `
    `/`
    ` shell the stylesheet expects around both. + +import { escapeHtml } from './text'; + +const KEYWORDS = + 'as|async|await|break|case|catch|class|const|continue|declare|default|delete|do|else|' + + 'enum|export|extends|false|finally|for|from|function|get|if|implements|import|in|' + + 'instanceof|interface|let|new|null|of|readonly|return|satisfies|set|static|super|' + + 'switch|this|throw|true|try|type|typeof|undefined|var|void|while|yield'; + +const CODE_RE = new RegExp( + [ + '(\\/\\/[^\\n]*|\\/\\*[\\s\\S]*?\\*\\/|#[^\\n]*)', // 1 comment + '(\'(?:\\\\.|[^\'\\\\])*\'|"(?:\\\\.|[^"\\\\])*"|`(?:\\\\.|[^`\\\\])*`)', // 2 string + '\\b(\\d[\\d_.]*[a-z]{0,2})\\b', // 3 number + `\\b(${KEYWORDS})\\b`, // 4 keyword + '\\b([A-Z][A-Za-z0-9_]*)\\b', // 5 type + '\\b([a-zA-Z_$][\\w$]*)(?=\\s*\\()', // 6 call + ].join('|'), + 'g', +); + +const CLASSES = ['tok-comment', 'tok-string', 'tok-number', 'tok-keyword', 'tok-type', 'tok-fn']; + +/** Token classes only — no AST, no grammar file. Good enough for the shapes we ship. */ +function highlightCode(source: string): string { + let out = ''; + let last = 0; + for (const match of source.matchAll(CODE_RE)) { + const at = match.index ?? 0; + out += escapeHtml(source.slice(last, at)); + const group = CLASSES.findIndex((_, i) => match[i + 1] !== undefined); + out += `${escapeHtml(match[0])}`; + last = at + match[0].length; + } + return out + escapeHtml(source.slice(last)); +} + +/** Terminal transcripts: prompt, command, pass/fail marks, and the 3-line error shape. */ +function highlightShell(source: string): string { + return source + .split('\n') + .map((line) => { + const prompt = /^(\s*)\$ (.*)$/.exec(line); + if (prompt !== null) { + return `${prompt[1]}$ ${escapeHtml(prompt[2] ?? '')}`; + } + const mark = /^(\s*)([✓✗])(.*)$/.exec(line); + if (mark !== null) { + const cls = mark[2] === '✓' ? 'tok-pass' : 'tok-fail'; + return `${mark[1]}${mark[2]}${escapeHtml(mark[3] ?? '')}`; + } + const code = /^(\s*)(X_[A-Z0-9_]+)(:.*)$/.exec(line); + if (code !== null) { + return `${code[1]}${code[2]}${escapeHtml(code[3] ?? '')}`; + } + const label = /^(\s+)(cause|fix|docs):(\s*)(.*)$/.exec(line); + if (label !== null) { + return `${label[1]}${label[2]}:${label[3]}${escapeHtml(label[4] ?? '')}`; + } + const diff = /^([+-])(.*)$/.exec(line); + if (diff !== null) { + const cls = diff[1] === '+' ? 'tok-added' : 'tok-removed'; + return `${escapeHtml(line)}`; + } + return escapeHtml(line); + }) + .join('\n'); +} + +const SHELL_LANGS = new Set(['bash', 'sh', 'shell', 'console', 'text', 'diff', 'yaml', '']); + +export function renderCode(lang: string, title: string | undefined, source: string): string { + const body = SHELL_LANGS.has(lang) ? highlightShell(source) : highlightCode(source); + const cls = SHELL_LANGS.has(lang) && lang !== 'yaml' ? ' class="terminal"' : ''; + const pre = `${body}
    `; + if (title === undefined) return pre; + return `
    ${escapeHtml(title)}${escapeHtml(lang)}
    ${pre}
    `; +} diff --git a/site/lib/html.ts b/site/lib/html.ts new file mode 100644 index 000000000..a7427e45a --- /dev/null +++ b/site/lib/html.ts @@ -0,0 +1,100 @@ +// The page fragments the layout and doc templates slot in: header nav, table of contents, +// previous/next pager, and the JSON-LD graph. + +import type { Page } from './config'; +import { ORIGIN } from './config'; +import { inline } from './markdown'; +import { escapeHtml } from './text'; + +export function navHtml(pages: readonly Page[], current: Page): string { + return ( + pages + // `menu: true` opts a page into the header; every page is still reachable from the + // footer and the pager, so the header never has to grow a scroll region on desktop. + .filter((page) => page.meta.menu === 'true') + .map((page) => { + const active = page.slug === current.slug ? ' aria-current="page"' : ''; + return `
  • ${escapeHtml(page.meta.nav ?? '')}
  • `; + }) + .join('\n') + ); +} + +export function tocHtml(headings: readonly { id: string; text: string }[]): string { + if (headings.length === 0) return ''; + const items = headings + .map((h) => `
  • ${inline(h.text)}
  • `) + .join('\n'); + return `
      \n${items}\n
    `; +} + +export function pagerHtml(pages: readonly Page[], index: number): string { + const previous = pages[index - 1]; + const next = pages[index + 1]; + if (previous === undefined && next === undefined) return ''; + const left = + previous === undefined + ? '' + : `Previous${escapeHtml(previous.meta.nav ?? previous.slug)}`; + const right = + next === undefined + ? '' + : `Next${escapeHtml(next.meta.nav ?? next.slug)}`; + return ` `; +} + +export function jsonLd(page: Page, isHome: boolean): string { + const graph = isHome + ? { + '@context': 'https://schema.org', + '@type': 'SoftwareApplication', + name: 'Ultimate', + applicationCategory: 'DeveloperApplication', + applicationSubCategory: 'Web framework', + operatingSystem: 'Linux, macOS', + description: page.meta.description, + url: `${ORIGIN}/`, + softwareVersion: '0.0.1', + license: 'https://opensource.org/licenses/MIT', + codeRepository: 'https://github.com/developerz-ai/ultimate', + runtimePlatform: 'Bun >= 1.3', + offers: { '@type': 'Offer', price: '0', priceCurrency: 'USD' }, + author: { '@type': 'Organization', name: 'developerz-ai' }, + } + : { + '@context': 'https://schema.org', + '@type': 'TechArticle', + headline: page.meta.title, + description: page.meta.description, + url: `${ORIGIN}${page.url}`, + dateModified: page.meta.updated, + inLanguage: 'en', + isPartOf: { '@type': 'WebSite', name: 'Ultimate', url: `${ORIGIN}/` }, + author: { '@type': 'Organization', name: 'developerz-ai' }, + proficiencyLevel: 'Expert', + }; + const crumbs = { + '@context': 'https://schema.org', + '@type': 'BreadcrumbList', + itemListElement: [ + { '@type': 'ListItem', position: 1, name: 'Ultimate', item: `${ORIGIN}/` }, + ...(isHome + ? [] + : [ + { + '@type': 'ListItem', + position: 2, + name: page.meta.title, + item: `${ORIGIN}${page.url}`, + }, + ]), + ], + }; + return [graph, crumbs] + .map( + (data) => + // `<` is escaped so no value can close the script element early. + ` `, + ) + .join('\n'); +} diff --git a/site/lib/markdown.ts b/site/lib/markdown.ts new file mode 100644 index 000000000..2ce84cede --- /dev/null +++ b/site/lib/markdown.ts @@ -0,0 +1,193 @@ +// The markdown renderer: inline spans and the block grammar the pages use — headings, lists, +// tables, fences, callouts, blockquotes and raw HTML — plus the h2 list the table of contents +// is built from. + +import { renderCode } from './highlight'; +import { escapeHtml, slugify } from './text'; + +export function inline(src: string): string { + return src + .split(/(`[^`]+`)/g) + .map((part) => { + if (part.length > 1 && part.startsWith('`') && part.endsWith('`')) { + return `${escapeHtml(part.slice(1, -1))}`; + } + return escapeHtml(part) + .replace(/\[([^\]]+)\]\(([^)\s]+)\)/g, '$1') + .replace(/\*\*([^*]+)\*\*/g, '$1') + .replace(/(^|[\s(])\*([^*\n]+)\*/g, '$1$2'); + }) + .join(''); +} + +const cells = (row: string): string[] => + row + .replace(/^\||\|$/g, '') + .split('|') + .map((c) => c.trim()); + +export interface Rendered { + readonly html: string; + readonly headings: readonly { id: string; text: string }[]; +} + +/** Block-level markdown: headings, lists, tables, fences, callouts, raw HTML. */ +export function markdown(src: string): Rendered { + const lines = src.split('\n'); + const headings: { id: string; text: string }[] = []; + const out: string[] = []; + let i = 0; + + const paragraph = (buffer: string[]): void => { + if (buffer.length > 0) out.push(`

    ${inline(buffer.join(' '))}

    `); + buffer.length = 0; + }; + + while (i < lines.length) { + const line = lines[i] ?? ''; + + if (line.trim() === '') { + i += 1; + continue; + } + + // fenced code + const fence = /^```(\S*)(?:\s+title="([^"]*)")?\s*$/.exec(line); + if (fence !== null) { + const buffer: string[] = []; + i += 1; + while (i < lines.length && !(lines[i] ?? '').startsWith('```')) { + buffer.push(lines[i] ?? ''); + i += 1; + } + i += 1; + out.push(renderCode(fence[1] ?? '', fence[2], buffer.join('\n'))); + continue; + } + + // ::: callout … ::: + const callout = /^:::\s*(ok|warn|info)(?:\s+(.*))?$/.exec(line); + if (callout !== null) { + const buffer: string[] = []; + i += 1; + while (i < lines.length && (lines[i] ?? '').trim() !== ':::') { + buffer.push(lines[i] ?? ''); + i += 1; + } + i += 1; + const label = callout[2] ?? callout[1] ?? 'note'; + const inner = markdown(buffer.join('\n')).html; + out.push( + ``, + ); + continue; + } + + // raw HTML block — passed through untouched, so the pitch can use the grid components + if (line.startsWith('<')) { + const buffer: string[] = []; + while (i < lines.length && (lines[i] ?? '').trim() !== '') { + buffer.push(lines[i] ?? ''); + i += 1; + } + out.push(buffer.join('\n')); + continue; + } + + // heading + const heading = /^(#{2,4})\s+(.*)$/.exec(line); + if (heading !== null) { + const level = (heading[1] ?? '##').length; + const text = heading[2] ?? ''; + const id = slugify(text); + if (level === 2) headings.push({ id, text }); + out.push( + `${inline(text)}` + + `#`, + ); + i += 1; + continue; + } + + // table + if (line.startsWith('|') && /^\|[\s:|-]+\|$/.test(lines[i + 1] ?? '')) { + const head = cells(line); + i += 2; + const rows: string[][] = []; + while (i < lines.length && (lines[i] ?? '').startsWith('|')) { + rows.push(cells(lines[i] ?? '')); + i += 1; + } + const thead = head.map((c) => `${inline(c)}`).join(''); + const tbody = rows + .map((row) => `${row.map((c) => `${inline(c)}`).join('')}`) + .join(''); + out.push( + `
    ${thead}${tbody}
    `, + ); + continue; + } + + // lists + const bullet = /^[-*]\s+(.*)$/.exec(line); + const ordered = /^\d+\.\s+(.*)$/.exec(line); + if (bullet !== null || ordered !== null) { + const tag = bullet !== null ? 'ul' : 'ol'; + const items: string[] = []; + while (i < lines.length) { + const current = lines[i] ?? ''; + const item = /^(?:[-*]|\d+\.)\s+(.*)$/.exec(current); + if (item !== null) { + items.push(item[1] ?? ''); + } else if (/^\s+\S/.test(current) && items.length > 0) { + items[items.length - 1] += ` ${current.trim()}`; + } else { + break; + } + i += 1; + } + out.push(`<${tag}>${items.map((t) => `
  • ${inline(t)}
  • `).join('')}`); + continue; + } + + // blockquote + if (line.startsWith('> ')) { + const buffer: string[] = []; + while (i < lines.length && (lines[i] ?? '').startsWith('>')) { + buffer.push((lines[i] ?? '').replace(/^>\s?/, '')); + i += 1; + } + out.push(`
    ${markdown(buffer.join('\n')).html}
    `); + continue; + } + + if (/^(---|\*\*\*)\s*$/.test(line)) { + out.push('
    '); + i += 1; + continue; + } + + // paragraph + const buffer: string[] = []; + while (i < lines.length) { + const current = lines[i] ?? ''; + if ( + current.trim() === '' || + current.startsWith('#') || + current.startsWith('|') || + current.startsWith('```') || + current.startsWith(':::') || + current.startsWith('<') || + current.startsWith('> ') || + /^(?:[-*]|\d+\.)\s/.test(current) + ) { + break; + } + buffer.push(current.trim()); + i += 1; + } + paragraph(buffer); + } + + return { html: out.join('\n'), headings }; +} diff --git a/site/lib/seo.ts b/site/lib/seo.ts new file mode 100644 index 000000000..1fdb30bf4 --- /dev/null +++ b/site/lib/seo.ts @@ -0,0 +1,34 @@ +// The SEO gate over the parsed pages: a title and a sized description on every page, and no two +// pages sharing either. It throws before anything renders, so a bad page fails the build. + +import type { Page } from './config'; + +export function seoCheck(pages: readonly Page[]): void { + const titles = new Map(); + const descriptions = new Map(); + for (const page of pages) { + const title = page.meta.title ?? ''; + const description = page.meta.description ?? ''; + if (title === '') throw new Error(`X_SEO_NO_TITLE: ${page.file} has no title in frontmatter`); + if (description === '') { + throw new Error(`X_SEO_NO_DESCRIPTION: ${page.file} has no description in frontmatter`); + } + if (description.length < 50 || description.length > 160) { + throw new Error( + `X_SEO_NO_DESCRIPTION: ${page.file} description is ${description.length} chars, needs 50-160`, + ); + } + const dupeTitle = titles.get(title); + if (dupeTitle !== undefined) { + throw new Error(`X_SEO_DUPLICATE_TITLE: ${page.file} repeats the title of ${dupeTitle}`); + } + const dupeDescription = descriptions.get(description); + if (dupeDescription !== undefined) { + throw new Error( + `X_SEO_DUPLICATE_DESCRIPTION: ${page.file} repeats the description of ${dupeDescription}`, + ); + } + titles.set(title, page.file); + descriptions.set(description, page.file); + } +} diff --git a/site/lib/text.ts b/site/lib/text.ts new file mode 100644 index 000000000..38993a92f --- /dev/null +++ b/site/lib/text.ts @@ -0,0 +1,35 @@ +// The text primitives every other stage builds on: HTML escaping, heading slugs, `{{var}}` +// template fills, and frontmatter parsing. + +export const escapeHtml = (s: string): string => + s + .replaceAll('&', '&') + .replaceAll('<', '<') + .replaceAll('>', '>') + .replaceAll('"', '"'); + +export const slugify = (s: string): string => + s + .toLowerCase() + .replace(/<[^>]+>/g, '') + .replace(/[^\w\s-]/g, '') + .trim() + .replace(/\s+/g, '-'); + +export const fill = (template: string, vars: Record): string => + template.replace(/\{\{(\w+)\}\}/g, (_, key: string) => vars[key] ?? ''); + +export function frontmatter(src: string): { meta: Record; body: string } { + if (!src.startsWith('---\n')) return { meta: {}, body: src }; + const end = src.indexOf('\n---', 4); + const meta: Record = {}; + for (const line of src.slice(4, end).split('\n')) { + const at = line.indexOf(':'); + if (at < 1) continue; + meta[line.slice(0, at).trim()] = line + .slice(at + 1) + .trim() + .replace(/^"(.*)"$/, '$1'); + } + return { meta, body: src.slice(end + 4).replace(/^\n+/, '') }; +} diff --git a/wiki/CLI-Reference.md b/wiki/CLI-Reference.md index c4ef4a6e8..52c5e1637 100644 --- a/wiki/CLI-Reference.md +++ b/wiki/CLI-Reference.md @@ -125,16 +125,21 @@ Errors: `X_DB_DRIFT`, `X_DB_GEN_FAILED`, `X_DB_MIGRATE_FAILED`, `X_DB_BRANCH_FAI ## x verify ```bash -x verify [--only step,step] [--skip step,step] [--json] +x verify [--json] ``` -The single gate. Green means shippable; CI runs exactly this. +The single gate. Green means shippable; CI runs exactly this. One step list, in cost order, shared +with the framework repo's own `bun run verify` — there is no `--only` and no `--skip`, because +"green" has to mean the same thing for everyone. A step with nothing to check in this project +reports as skipped (`-`), never as passed. | Step | Checks | |---|---| | `typecheck` | `tsc` across every workspace | | `lint` | Biome: no `any`, no default exports, no bare `Error`, no raw colours, no hardcoded user-facing strings | -| `boundaries` | surface and layer imports, resolved transitively | +| `boundaries` | surface and layer imports, resolved transitively; package tiers in a monorepo | +| `filesize` | a source file over 500 lines | +| `package-shape` | a workspace package missing `README.md`, `CLAUDE.md`, `tsconfig.json`, `src/index.ts` | | `unit` | pure logic — services, money, policy predicates, matchers | | `contract` | action/query schemas, policy denials, emitted OpenAPI and MCP shapes | | `live` | live-query snapshot, incremental patches, reconnect delta, policy-filtered rows | @@ -146,12 +151,17 @@ The single gate. Green means shippable; CI runs exactly this. | `budgets` | per-route JS bytes and LCP | | `manifest` | `x.manifest.json` freshness | +A test's type is its filename suffix — `*.contract.test.ts`, `*.live.test.ts`, `*.job.test.ts`, +`*.e2e.test.ts` (or any test under `e2e/`), `*.eval.test.ts`. Everything else is a unit test, so no +test can fall between two steps. + ```bash -$ x verify --only budgets --json -{"ok":false,"checks":[{"name":"budgets","ok":false,"failures":[ - {"route":"site/pricing","metric":"js","actual":"61kb","limit":"40kb", - "cause":"chart.js via shared/ui/button.tsx", - "fix":"x fix boundary site/pricing/page.tsx"}]}]} +$ x verify --json +{"ok":false,"command":"verify","summary":"1 of 15 steps failed","steps":[ + {"name":"budgets","ok":false,"durationMs":812,"skipped":false,"findings":[ + {"code":"X_BUDGET_EXCEEDED","cause":"site/pricing ships 61kb of JS, over the 40kb budget", + "fix":"x fix boundary site/pricing/page.tsx", + "docs":"https://ultimate.dev/errors/X_BUDGET_EXCEEDED","at":"site/pricing"}]}]} ``` Errors: `X_VERIFY_FAILED` (with the failing step names), plus each step's own code. diff --git a/wiki/Contributing.md b/wiki/Contributing.md index 9fe1a7d0e..19ce9295b 100644 --- a/wiki/Contributing.md +++ b/wiki/Contributing.md @@ -107,9 +107,9 @@ TypeScript strictness comes from [`tsconfig.base.json`](https://github.com/devel | Task | Command | |---|---| -| all tests | `bun test` | +| all tests | `bun run test` (bare `bun test` also scans `examples/`, which has its own gate) | | one file | `bun test packages/core/src/errors.test.ts` | -| typecheck the whole graph | `bun run typecheck` (`tsc -b --pretty`) | +| typecheck the framework graph | `bun run typecheck` (`tsc -b --pretty`) | | clean the build info | `bun run typecheck:clean` | | lint + format check | `bun run lint` (`biome check .`) | | autofix | `bun run lint:fix` / `bun run format` | diff --git a/wiki/Error-Codes.md b/wiki/Error-Codes.md index 1b4ac17b8..27777c57f 100644 --- a/wiki/Error-Codes.md +++ b/wiki/Error-Codes.md @@ -260,10 +260,12 @@ X_DB_DRIFT: schema differs from migrations | `X_CLI_UNKNOWN_COMMAND` | not a command | a typo | `x help` — the suggestion is in `fix` | | `X_CLI_BAD_FLAG` | flag rejected | unknown flag, or a bad value | `x --help` | | `X_CLI_UNEXPECTED` | the CLI itself failed | a bug, or a broken environment | `x doctor --json` and attach it to an issue | -| `X_VERIFY_FAILED` | one or more verify steps failed | the gate is red | `x verify --only --json` for that step alone | +| `X_VERIFY_FAILED` | one or more verify steps failed | the gate is red | `x verify --json` — every step's findings arrive in one run | | `X_TYPECHECK_FAILED` | `tsc` failed | a type error anywhere in the workspace | `bunx tsc -b --pretty false` | | `X_LINT_FAILED` | Biome failed | `any`, a default export, a bare `Error`, a raw hex colour | `bunx biome check --write .` | -| `X_TEST_FAILED` | a test type failed | a red test | `bun test --test-name-pattern ""` | +| `X_TEST_FAILED` | a test type failed | a red test | the `fix` is the exact `bun test …` invocation the step ran | +| `X_FILE_TOO_LONG` | a source file is over 500 lines | one file doing several jobs | split it; the `fix` names the file | +| `X_PACKAGE_SHAPE` | a workspace package is missing a contract file | a package added by hand | `bun run scripts/new-package.ts --only ` | | `X_CONTRACT_BREAKING` | the OpenAPI contract broke | a required input added, or an operation removed | give the input a `.default()`, or bump the package version | | `X_BUILD_FAILED` | `x build` failed | a static check or the bundler | read `cause`; the failing step is named | | `X_DEPLOY_FAILED` | a deploy step failed | the compose/helm command exited non-zero | run the printed command directly for full output | diff --git a/wiki/Policies-And-Authz.md b/wiki/Policies-And-Authz.md index 02ba26856..cd80c9e8b 100644 --- a/wiki/Policies-And-Authz.md +++ b/wiki/Policies-And-Authz.md @@ -102,7 +102,7 @@ $ x policy explain publishPost --json |---|---|---| | one path explained | `x policy explain --json` | `policies.list` | | every policy + its users | `x policy list --json` | `policies.list` | -| unprotected surfaces | `x verify --only boundaries --json` | `budgets.report` / `manifest.get` | +| unprotected surfaces | `x verify --json` (the `boundaries` step) | `budgets.report` / `manifest.get` | The `branches` array is what the generated tests enumerate — one denial test per branch. An untested branch is a red build.