[immakermatty]

Why

Every pull request ran upstream's whole vitest suite: about 900 test files, 45 minutes of serial tests (fileParallelism: false), four shards. The last pull request run before this change (PR #22, run 37213822983) took 18.6 min to green and 58 billed minutes; its slowest shard took 17.5 min, because vitest's path-hash split put delta-context.e2e (242 s), direct-coordination.e2e (185 s) and group-local-vm.e2e (113 s) into one shard. The overlay's own tests take about 20 s.

Target (plan DEV-6649, M6): a typical pull request is green in about 3 minutes; main still runs the whole suite after every merge.

What changes

• Pull requests that change only Lazurio files, documentation or tests run what the change can affect. The overlay job (the required check Overlay, typecheck and lint, name unchanged) runs scripts/lazurio-ci-scope.mjs. It hands the four vitest shards the overlay's own tests (every test file the overlay adds or changes against UPSTREAM_SHA) plus the files the pull request changes (merge-base diff, renames off), and the shards run vitest related on them: vitest's own import graph picks the changed test files and every test that imports a changed file.

• The whole suite runs on pushes to main, on manual runs (the dispatched check of a rebuild on a new upstream tag), and on a pull request that changes:

  • upstream code, i.e. any file present at UPSTREAM_SHA other than Markdown and test files. Measured reason: no test imports server/index.ts (vitest related server/index.ts selects 0 files) because about a hundred upstream end-to-end tests start it as a child process, and the UI end-to-end tests load the app, index.html included, through a Vite dev server. Import-graph selection would silently skip them;
  • .github/ (CI, the upstream pin UPSTREAM_TAG/UPSTREAM_SHA, the allowlist), the selection script or the shard config, any package.json, the lockfile or workspace file, any vite/vitest config, a tsconfig, or shared test helpers and fixtures (testing/, test/, __mocks__/, *fixtures/, which includes the vitest setup file).

  It also runs when the pull request's merge base does not stand on the pinned upstream commit, and whenever the selection fails (missing base, invalid SHAs, unreadable diff, odd path). It fails closed, with a warning in the log. In a shard, a related scope with an empty or unreadable list fails the job instead of running nothing.

• Shards balanced by measured duration. lazurio/vitest.config.mjs takes upstream's vite.config.ts unchanged and adds a vitest sequencer (sequence.sequencer, vitest's standard extension point). Its shard() places files longest-first onto the least-loaded shard, deterministically. Durations: lazurio/vitest-durations.json, the 95 files that took 5 s or more on average in runs 37476583884 and 37478790478; every other file weighs 0.7 s. A listed file that no longer exists fails the run (rename or drop the entry; the overlay job's node test checks it too). fileParallelism: false stays.

• Broker and Electron unit tests still run in shard 1 in both modes (about 30 s).

• Runbook (docs/operations/lazurio-fork-release.md, "Testing before a release") describes the selection, its limit and how to see what a pull request runs (UPSTREAM_SHA=<pin> node scripts/lazurio-ci-scope.mjs <base> <head>).

What does not change

Release contract, UPSTREAM_TAG/UPSTREAM_SHA, tags, main history, the overlay guard and its allowlist, lazurio-release.yml, lazurio-archive.yml, the required check name and the vitest job names, fileParallelism. No upstream file changes, so there is no new allowlist entry. main runs exactly what it ran before (the whole suite, broker and Electron tests, archive), only split differently across the four shards.

Deleted tests

All in the Lazurio-owned scripts/lazurio-release-contract.node-test.mjs. Upstream tests are not touched; the overlay may not change them.

File	Test	Reason
scripts/lazurio-release-contract.node-test.mjs	the runbook and fork contract name the operative facts	Only checks that the runbook and LAZURIO.md contain certain words (documentation wording).
scripts/lazurio-release-contract.node-test.mjs	CI is read-only and pins the exact upstream base: the pnpm typecheck and vitest run --shard= assertions	They pin the CI layout. The rest of that test stays: read-only permissions, the pin agreeing with the archive version, and the upstream-workflow gate.

Kept on purpose: the upstream-workflow gate assertion (it guards a publishing boundary), the allowlist checks, and the shell-slot checks (the only rebuild guard of the slot).

Verification

CI on this pull request (it changes CI, so every run is the whole suite):

Run	Head	To green	Shards (job min)	Billed min
before: 37213822983 (PR #22)	1a2e073	18.6 min	9.3 / 13.5 / 11.4 / 17.5	58
before: 37222187977 (push to main)	a57f8dc	18.8 min	9.2 / 13.9 / 11.1 / 17.3	58
37476583884	40cbe95	14.6 min	13.2 / 10.4 / 13.0 / 13.2	56
37478790478	0c86eae	14.7 min	13.3 / 10.1 / 13.0 / 13.3	56
37481122697	1901846 (this head)	15.4 min	13.6 / 12.4 / 12.8 / 14.1	59

On the first two runs one shard finished about three minutes early: the first durations came from an older run in which several files were slower (direct-coordination.e2e 186 s, now 142 s). Re-measuring on those two runs narrowed the vitest time per shard from 587–765 s to 717–813 s. The whole run on this head was about 6 % slower (3 028 s of tests against 2 850 s), which is runner variance. A whole-suite run costs as many minutes as before and is about 3–4 minutes faster.

The related path in CI: throwaway PR #26, stacked on this branch, changed only a comment in the Lazurio module server/github-intake.ts. Run 37479124169 was green in 2.3 min, 8 billed minutes (overlay 1.2 min, archive 1.4 min, shards 0.7–1.0 min). It selected the 12 overlay test files (3/1/4/4 per shard), and shard 1 also ran broker + Electron. PR #26 is closed and its branch deleted.

Replay of real history (UPSTREAM_SHA=<pin of that head> node scripts/lazurio-ci-scope.mjs <base> <head>; the selected files come from vitest's graph on the current tree):

• Merged PRs: docs: Disable the upstream workflows a first release registers (#6) #7, release: Ship the Steward team file inside the release archive (#4) #8, docs: Record the Auto approval level, the requested-scope start, and the first-release npm-package step (#3, #5, #6) #9, docs: Lazurio MausBot naming and the Environment's Composio for bots (DEV-6632) #12, docs: LAZURIO.md keeps upstream branding and records the shell loader #17 and Docs: the Lazurio rail takes padding, not margin; one allowlist line per file #18 (docs, release script, contract test) would run related: only the overlay tests, 2–12 files, 3–21 s of tests. GitHub intake and Steward team (DEV-6632 M2) #2, server: OMB_HEADLESS_FULL_ACCESS, Full access over the API as an operator opt-in (#3) #10, Live browser: real page size for clicks, and wait while the bot uses the browser (#13, #14) #15, Lazurio shell slot: rail, column head and Buddy from /.lazurio/shell.js (#16) #20 and lazurio: the shell takes the skin's colours through its roles #22 run the whole suite because they edit the workflow, mostly to add allowlist entries. Lazurio distribution: release pipeline on v0.1.91 and OMB_DEFAULT_BOT_CWD (DEV-6632 M1) #1 and Rebuild the Lazurio overlay on upstream v0.1.95 #19 (bootstrap and rebuild) were dispatched runs, so the whole suite.
• The 28 overlay commits on main: 16 related (2–12 test files, 3–21 s), 9 whole suite for workflow changes and 3 for upstream code (scripts/control-omb.ts twice, server/browser-engine.ts). Under pure vitest related, those 3 would have selected 46–49 % of the suite while still missing the server's child-process tests.

Locally (macOS, Node 26, pnpm 10.33.0): node --test scripts/lazurio-release-contract.node-test.mjs scripts/lazurio-ci-scope.node-test.mjs 14/14 pass; pnpm lint and pnpm typecheck clean; pnpm broker:test and pnpm test:electron pass; vitest related --run --config lazurio/vitest.config.mjs --shard=N/4 on a docs-only diff ran the 12 overlay tests, 4–12 s per shard. The duration split over the real 922 test files puts each file in exactly one shard (checked through the vitest Node API). I did not run the whole vitest suite on macOS; the three CI runs above ran it on Linux.

Risks and follow-ups

• A selected pull request (Lazurio files, docs, tests) can still miss a test that reads a Lazurio file from disk, or a Lazurio module that only the spawned server loads. The overlay's own e2e tests boot the real server on every pull request, and main runs everything after each merge; repair forward (plan risk note).
• A pull request that only adds an entry to allowed_upstream_changes runs the whole suite, because the allowlist lives in the workflow file. Moving it into its own file would let such pull requests run selectively, but that touches the release contract, so I left it as a possible follow-up.
• lazurio/vitest-durations.json drifts as the suite changes. Drift only unbalances shards; a rebuild that renames a listed file fails until the entry is updated (described in the runbook and the config).

Refs #24

🤖 Generated with Claude Code

Note

Run scope-aware Vitest on pull requests and balance shards by test duration

• Adds scripts/lazurio-ci-scope.mjs, which resolves pull-request scope from the Git diff against the pinned upstream commit and publishes a full or related Vitest scope plus a related-file list to GitHub Actions outputs. Local runs can print the same decision from explicit base and head commits.
• PRs that only change overlay code, docs, or tests now run Vitest in related-test mode with the selected files; upstream code changes and files outside the import graph select the full suite.
• Adds a duration-based shard sequencer in lazurio/vitest.config.mjs using measured weights from lazurio/vitest-durations.json, replacing Vitest's default path-based shard assignment.
• Updates the fork CI workflow to fetch the pull-request base, run the scope script, and run Vitest through the new shard-aware script; broker and Electron tests remain on shard 1.
• Adds a full Node test suite for scope selection, Git integration, duration splitting, and duration-data validation, and updates the release-contract tests for the new command structure.
• Risk: readDurations now throws when a duration in lazurio/vitest-durations.json is invalid or a listed test file no longer exists; the runbook requires updating the duration list when a listed test is renamed or removed.

^{Macroscope summarized 1901846.}

[agentrozjedemeai]

Exact-head QA at 1901846 (DEV-6649): APPROVED. I reviewed the PR diff, scope fallback and duration sequencer, and replayed scope selection on this base/head (full suite as CI changed). The isolated checkout passed all 14 Node contract/selection tests; diff check was clean. The six required checks on this head are green; the comment-only PR #26 exercised the related path successfully. Known limit remains: import-graph selection may miss external file reads/child-process loads of overlay code; overlay e2e runs on PRs and main runs the full suite.