Skip to content

feat(dev): local candidate build/validate/promote loop with rollback (rail 3.8.53, #8084 local side) - #16180

Merged
diegosouzapw merged 3 commits into
release/v3.8.52from
feat/rail-candidate-dev-loop
Oct 10, 2026
Merged

diegosouzapw merged 3 commits into
release/v3.8.52from
feat/rail-candidate-dev-loop

Conversation

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Summary

This is the local side of RFC #8084 (rail 3.8.53, task 5), exposed as npm run dev:candidate (scripts/dev/candidate.mjs). It builds one candidate artifact, validates the packaged runtime of that exact artifact, and promotes the same artifact with directory renames. If the promoted slot then fails its health check, it rolls back automatically. It never touches an existing OmniRoute install: every write goes under the gitignored _artifacts/candidate/, unless the operator passes --target.

Refs #8084

What changed

  • scripts/dev/candidate.mjs provides these subcommands. All of them accept --dry-run and --json.
    • build: runs npm pack on the current tree (requires dist/server.js) or takes --from-tarball <file>, then runs npm install -g --prefix into _artifacts/candidate/<id>/prefix. A plain tar -x cannot boot because the tarball has no node_modules ([BUG] npm global install on macOS Apple Silicon: vendored dist/node_modules/better-sqlite3/prebuilds/ missing darwin-arm64.node #11242); this is the same path check:pack-boot uses. The id is the short HEAD sha (-dirty when the tree has local changes) or tgz-<sha256>. A clean id that already exists is reused (build-once); --force rebuilds it.
    • validate: boots node <slot>/…/bin/omniroute.mjs serve --port <free port> with a fresh DATA_DIR, an isolated HOME/XDG/APPDATA, and fake secrets. It removes the operator's OMNIROUTE_API_KEY/ROUTER_API_KEY/STORAGE_ENCRYPTION_KEY from the environment. GET /api/health must return 200 and GET /v1/models must return 200 with a list. The process group is stopped afterwards, and the verdict goes to validation.json, bound to the tarball sha256.
    • promote: performs the renames previous → discard, current → previous, <id> → current. If any rename fails, the renames already applied are reversed. It refuses to promote unless there is a passing validation for this tarball hash (--force overrides). --target <dir> selects another active slot; the destructive or overlapping targets /, home, the repo and the candidate root are refused.
    • rollback: swaps current and previous.
    • run: build → validate → promote → re-validate the promoted slot → rollback if that check fails. A candidate that fails the first validation is never promoted. On a first promotion into an empty slot, a failure demotes the artifact back instead of rolling back.
  • scripts/check/check-pack-boot.mjs: now exports stopChild so the loop reuses the same process-group shutdown instead of duplicating it. pickTarball is reused as well. No behavior change.
  • package.json: adds "dev:candidate": "node scripts/dev/candidate.mjs".
  • Docs: new section "Local candidate loop" in docs/ops/CONTRIBUTION_GOLDEN_PATH.md, plus an optional smoke line under Artifact Validation in docs/ops/RELEASE_CHECKLIST.md.
  • Hard Rule deps: bump qs from 6.14.1 to 6.14.2 #13: only execFileSync/spawn with argument arrays; values go through env. A static test pins this (no shell: true, no exec(/execSync(, no os.tmpdir() or /tmp).

Found by the real run (and fixed in bd1c6442dd)

The first real validate against the published omniroute@3.8.51 tarball booted and returned 200 from /api/health, but /v1/models returned 401. Validation correctly refused to promote, and the active slot was untouched. There were two causes, both in how the packaged CLI loads .env files (it fills any key that is still undefined):

  1. <packageRoot>/.env, which postinstall copies from .env.example, contains INITIAL_PASSWORD=CHANGEME. Any password counts as a credential surface for the model catalog (src/app/api/v1/models/catalogRequest.ts). Fix: pin INITIAL_PASSWORD="" (set but falsy) and OMNIROUTE_CLI_SKIP_REPO_ENV=1.
  2. The CLI also reads <default data dir>/.env, which is derived from HOME/XDG_CONFIG_HOME/APPDATA. That is the operator's real ~/.omniroute/.env. Fix: an isolated HOME and XDG/APPDATA under the slot.

I checked the isolation empirically: a snapshot of ~/.omniroute (paths, mtimes, sizes) taken before and after a full validate boot shows no changes.

Validation

Gate Command Result
New unit tests node --import tsx/esm --test tests/unit/dev-candidate-loop.test.ts 32/32 pass (TDD: RED before each implementation step)
Reused gate tests node --import tsx/esm --test tests/unit/check-pack-boot.test.ts tests/unit/pack-boot-runtime-paths.test.ts pass (50/50 combined with the new file)
Real end-to-end npm run dev:candidate -- run --from-tarball omniroute-3.8.51.tgz --json (published tarball, real npm install, real boot) exit 0: validate /api/health 200 + /v1/models 200 → promoted → post-promotion re-check 200/200 (the relocated npm prefix still boots)
Real failure path same run before the .env fix exit 1, stage: "validate", /v1/models: HTTP 401; active slot untouched
Real rollback guard npm run dev:candidate -- rollback with no previous slot exit 2, "nothing to roll back to"
Real dry-run npm run dev:candidate -- run --dry-run --json exit 0, plan only, nothing written
Isolation snapshot diff of ~/.omniroute around a real validate no changes
ESLint npx eslint tests/unit/dev-candidate-loop.test.ts (scripts/** is eslint-ignored by config) 0 problems
Typecheck npm run typecheck:core exit 0
Dead code npm run check:dead-code OK: 369 (baseline 377); knip ignores scripts/**
Test discovery node scripts/check/check-test-discovery.mjs OK (test lives at tests/unit/dev-candidate-loop.test.ts, which is collected)
Docs npm run check:doc-links / npm run check:fabricated-docs PASS / no fabricated references
Tracked artifacts npm run check:tracked-artifacts OK

Not run locally, per the campaign rules (shared devbox): the full unit suite and npm run build. With no local dist/, the real run used --from-tarball with the published 3.8.51 package; the npm pack branch of build is covered by unit tests (plan and precondition) only.

⚠️ base-red inherited: #15306

Add scripts/dev/candidate.mjs (npm run dev:candidate), the local side of
RFC #8084: build one candidate from the current tree (npm pack) or a
prebuilt tarball into an isolated npm prefix under _artifacts/candidate/<id>,
boot that packaged CLI on a free port with an isolated DATA_DIR and fake
secrets, require /api/health and /v1/models to answer 200, promote the same
validated artifact with compensated renames, and roll back automatically
when the promoted slot fails its health check. Every command supports
--dry-run and --json; child processes use argument arrays only (Hard Rule #13).

stopChild is now exported from check-pack-boot.mjs so the loop reuses the
same process-group shutdown instead of duplicating it.
The first real validate run against the published 3.8.51 tarball booted and
answered /api/health, but /v1/models returned 401: the packaged CLI fills
undefined keys from <packageRoot>/.env (postinstall copies .env.example with
INITIAL_PASSWORD=CHANGEME), and any password is a credential surface for the
model catalog. It also reads the default data dir's .env, derived from
HOME/XDG_CONFIG_HOME/APPDATA, i.e. the operator's real ~/.omniroute/.env.

The candidate now gets an isolated HOME (plus XDG/APPDATA) under its slot,
INITIAL_PASSWORD pinned to an empty value, a fresh DATA_DIR on every
validation, and --log so failures show server output. run --dry-run no
longer claims the candidate is active.
@diegosouzapw
diegosouzapw merged commit cff1461 into release/v3.8.52 Oct 10, 2026
23 of 46 checks passed
diegosouzapw added a commit that referenced this pull request Oct 10, 2026
Brings in #16073 (RELEASE_STRATEGY.md, ROADMAP See-also), #16180
(dev:candidate, RELEASE_CHECKLIST, meta.json), #16172 (dist-tag.mjs and the
dormant LTS workflows) and #16066 (Mergify checks_timeout). package.json keeps
both release:dry-run-lts-cut and labels:sync.

Now that the base carries them, the LTS-cut dry-run uses the real
scripts/release/dist-tag.mjs resolver (the built-in fallback is gone), treats a
dormant workflow missing at the cut source as NOT READY, and the runbook /
GO-NO-GO docs link RELEASE_STRATEGY.md and name the real forward-port secret
and nightly publish variable. The G11 fixture is the real .mergify.yml.
@github-actions

Copy link
Copy Markdown
Contributor

CI Coverage Report

  • Coverage job: skipped
  • PR test policy: success

Coverage artifact was not available for this run.

@diegosouzapw
diegosouzapw deleted the feat/rail-candidate-dev-loop branch October 16, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant