Skip to content

feat(ops): canary deploy with provenance gate, real smoke and rollback anchor - #10446

Merged
diegosouzapw merged 1 commit into
release/v3.8.50from
feat/deploy-canary
Aug 15, 2026
Merged

diegosouzapw merged 1 commit into
release/v3.8.50from
feat/deploy-canary

Conversation

@diegosouzapw

Copy link
Copy Markdown
Owner

Closes #10429. Completes the trio opened after the 2026-08-14 gateway outage (#10427 provenance, #10428 DATA_DIR guard, this one).

Why

Deploying the internal gateway was a manual sequence — build, pack, scp, npm i -g, pm2 restart — with nothing recording what landed and nothing proving the new build served traffic.

On 2026-08-14 that produced a silent outage: the installed package had been built from a branch predating #10373, the process came up, /api/monitoring/health answered healthy, and every request returned 502 … Executor result must contain a Response until a human hit it. A health check could never have caught it — only a real completion exercises the egress path where that defect lived.

What changed

scripts/ops/deployCanary.ts — the policy, as pure functions:

  • planCanaryDeploy delegates to fix(build): validate packaged BUILD_SHA provenance + expose buildSha on health #10427's resolveBuildProvenance, so the pack gate and the deploy path cannot disagree about what "shippable" means.
  • evaluateSmoke grades health plus completion probes. Zero probes is a failure: "no probe ran" must never read as "everything is fine" — that is exactly how a broken egress path hides behind a green health check.
  • buildRemoteSteps emits argv arrays, never shell strings (Hard Rule deps: bump qs from 6.14.1 to 6.14.2 #13), ordered so the rollback anchor is captured before the install overwrites it and the SHA is verified only after the restart.

scripts/ops/deploy-canary.mjs — the thin shell that performs the side effects, supports --dry-run, requires at least one --model, and prints the rollback command (with the previous SHA) when the smoke fails. The completion probe requires a parseable body with choices, not just a 2xx — the outage returned failures inside otherwise-normal responses.

Validation

  • 10 unit tests, including the incident as a case: healthy process + one broken completion → fail + rollback, plus the empty-probe-list guard and the argv/no-shell-metacharacter assertion.
  • End-to-end --dry-run exercised both paths: refused an untraceable artifact, then printed the full plan for a valid one (BUILD_SHA a36fbdcc8d is on the release line).
  • Gates clean: typecheck:core, check:cycles, check:docs-all, check-env-doc-sync, check-dead-code (249 vs baseline 415), validate-pack-artifact --policy-only, eslint on touched files.

⚠️ base-red inherited: #9985

…k anchor

Deploying the internal gateway was a manual build/pack/scp/npm-i/pm2-restart
sequence with no record of what landed and no proof it served traffic. On
2026-08-14 that shipped a package built from a branch predating #10373: the
process came up, health said 'healthy', and every request returned 502 until a
human hit it.

- scripts/ops/deployCanary.ts holds the policy as pure functions: refuse an
  artifact that is not traceable to the release line (reuses #10427), and grade
  the deploy on health PLUS at least one real completion. Zero probes fails —
  'no probe ran' must never read as 'everything is fine', which is exactly how a
  broken egress path hides behind a green health check.
- Remote steps are emitted as argv arrays, never shell strings (Hard Rule #13),
  ordered so the rollback anchor is captured before the install overwrites it.
- scripts/ops/deploy-canary.mjs performs the side effects, supports --dry-run,
  and prints the rollback command when the smoke fails.

Closes #10429
@diegosouzapw
diegosouzapw merged commit 1287b6a into release/v3.8.50 Aug 15, 2026
21 checks passed
@diegosouzapw
diegosouzapw deleted the feat/deploy-canary branch August 15, 2026 06:24
tkgo11 pushed a commit to tkgo11/OmniRoute that referenced this pull request Sep 23, 2026
…k anchor (diegosouzapw#10446)

Deploying the internal gateway was a manual build/pack/scp/npm-i/pm2-restart sequence with no record of what landed and no proof it served traffic. On 2026-08-14 that shipped a package built from a branch predating diegosouzapw#10373: the process came up, health said 'healthy', and every request returned 502 until a human hit it.

scripts/ops/deployCanary.ts holds the policy as pure functions — refuse an artifact that is not traceable to the release line (reusing diegosouzapw#10427), and grade the deploy on health PLUS at least one real completion. Zero probes fails: 'no probe ran' must never read as 'everything is fine', which is exactly how a broken egress path hides behind a green health check. Remote steps are argv arrays, never shell strings (Hard Rule diegosouzapw#13), ordered so the rollback anchor is captured before the install overwrites it.

scripts/ops/deploy-canary.mjs performs the side effects, supports --dry-run, and prints the rollback command when the smoke fails.

Closes diegosouzapw#10429
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…k anchor (diegosouzapw#10446)

Deploying the internal gateway was a manual build/pack/scp/npm-i/pm2-restart sequence with no record of what landed and no proof it served traffic. On 2026-08-14 that shipped a package built from a branch predating diegosouzapw#10373: the process came up, health said 'healthy', and every request returned 502 until a human hit it.

scripts/ops/deployCanary.ts holds the policy as pure functions — refuse an artifact that is not traceable to the release line (reusing diegosouzapw#10427), and grade the deploy on health PLUS at least one real completion. Zero probes fails: 'no probe ran' must never read as 'everything is fine', which is exactly how a broken egress path hides behind a green health check. Remote steps are argv arrays, never shell strings (Hard Rule diegosouzapw#13), ordered so the rollback anchor is captured before the install overwrites it.

scripts/ops/deploy-canary.mjs performs the side effects, supports --dry-run, and prints the rollback command when the smoke fails.

Closes diegosouzapw#10429
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(ops): reproducible canary deploy script with provenance check, smoke and rollback

2 participants