ci: e2e-staging-saas on staging + canary auto-issue thresholded at 3 reds

[HongmingWang-Rabbit]

Summary

Two CICD-review quick wins consolidated into one PR. Closes items 5 and 10 from the comprehensive review.

1. e2e-staging-saas fires on staging, not just main

Full-lifecycle SaaS E2E was main-only, so it caught regressions after they shipped to staging (and into the auto-promote PR). Now fires on staging push + PR too — catches them before the staging→main promotion opens, making the canary→auto-promote chain meaningfully more trustworthy.

`paths-filter` unchanged → blast radius stays the same (only provisioning-critical changes trigger the ~25-35 min run).

2. Canary auto-issue thresholded at 3 consecutive failures

The 30-min canary was opening "🔴 Canary failing" issues on every red. Transient flakes (CF DNS hiccup, AWS blip) generated noise — see #2090's history this session.

Now:

• First failure → look up the prior `THRESHOLD-1` runs of the same workflow
• Open an issue only when ALL of those ALSO failed (i.e. this is the 3rd consecutive red, ~90 min of sustained failure)
• If issue already open → continue commenting per-failure so the streak stays visible
• Auto-close on next green (unchanged)

Threshold rationale: canary fires every 30 min → 3 reds = ~90 min of sustained failure. Past any single-run flake but well inside the deploy window so a real outage still surfaces fast.

Out of band — already done in this CICD-review pass:

• ✅ Item 1: AWS `Environment=production` tag backfilled on the 3 legacy tenants
• ✅ Item 3: `staging` branch protection now requires 1 review (parity with main)

Test plan

• YAML parses
• CI green on this PR
• Auto-merge on green

🤖 Generated with Claude Code

[HongmingWang-Rabbit]

Skipping review: required check E2E Staging SaaS failing. Failure was at TLS/DNS propagation step at 19:21Z, during the lib/pq cached-prepared-statement window before CP #282 deployed (~20:30Z). Re-triggered the failed job — should pass now. Will re-evaluate once green.

[HongmingWang-Rabbit]

Skipping review this pass: required check E2E Staging SaaS still failing at TLS/DNS propagation step (Could not resolve host: e2e-20260426-...staging.moleculesai.app after 900s wait). Looks like a Cloudflare/cloudflared propagation issue on staging rather than PR code — same symptom across #2108/#2096/#2064 in the same window. No action required on this PR; will re-evaluate when staging infra is healthy.

[HongmingWang-Rabbit]

Re-triggered failed CI (CP #284 just merged at 21:49Z, fixing the silent-DNS-failure root cause that was stranding tests at tenant TLS / DNS propagation timed out). Re-evaluate once the new run completes.

[HongmingWang-Rabbit]

Still skipping: failed E2E correlates with the active Cloudflare Minor Service Outage (status page indicator updated 22:08:40Z). CP #284's fast-fail surfaces CF API failures in ~16s instead of 900s — that's working as intended; the underlying CF outage is the blocker. Re-trigger when cloudflarestatus.com returns to operational.

[HongmingWang-Rabbit]

Code review (5-axis)

Correctness: ✓ Two unrelated changes bundled here:

1. canary-staging.yml — issue-on-failure now thresholded at 3 consecutive reds via listWorkflowRuns lookback. Existing-issue path is preserved (still comments on subsequent fails). Logic looks right: runs.workflow_runs.slice(0, threshold-1).filter(r => r.id !== context.runId) correctly excludes the current run from the prior-runs check.

2. e2e-staging-saas.yml — adds staging to push branches. Now catches regressions before they reach the staging→main promote PR.

Tests: N/A — workflow YAML, no test surface. Inline comment block traces the "why" sufficiently.

Architecture: Two unrelated changes in one PR is a minor splitting violation per the skill's "separate refactoring from feature work" guideline, but they're both small and operationally tied (CI alerting tuning). Acceptable bundle.

Security: ✓ No new secrets, no new permissions.

Performance: ✓ One extra listWorkflowRuns API call per failed canary. Trivial.

Optional

• The WORKFLOW_PATH env injection comment notes that context.workflow is the workflow name not the file path — that's a CI footgun worth a top-of-file note in the workflow's own header so the next maintainer doesn't re-trip on it. Not blocking.
• Companion to fix(ci): canary teardown safety-net slug pattern (was reversed) #2125 (the slug-pattern fix) — once both land, the canary cancellation-cleanup and noise-floor stories are both closed. Worth referencing fix(ci): canary teardown safety-net slug pattern (was reversed) #2125 in the merge commit if the merge-queue squashes.

CI failure on E2E Staging SaaS is the pre-CF-zone-cleanup symptom (failure timestamp 21:52Z, before 22:50Z cleanup). Re-trigger should pass — already kicked.

LGTM modulo the failing CI.

[HongmingWang-Rabbit]

Skip this pass: required check E2E Staging SaaS still failing — but the failure mode shifted post-CF-zone-cleanup. The feature test now passes all 11 steps cleanly (provisioning, A2A roundtrip, HMA memory, peer discovery, delegation all green at 23:03Z). The new failure is in the teardown-verify step:

[23:04:04] Teardown returned non-2xx (may already be gone)
⚠️  LEAK: org e2e-20260426-24964590920-4 still present post-teardown (count=1)
##[error]Process completed with exit code 4

The DELETE on /cp/admin/tenants/<slug> is non-2xx and the org row persists. Possibly a regression surface from CP #287's switch to fail-fast on cascade failures (formerly log-and-continue) — if any single step in the purge cascade returns an error, the whole DELETE fails and the org isn't marked purged. Requires CP staging logs from 23:04:04Z to pinpoint which cascade step is erroring.

Not blocking this PR's merit (workflow tweaks are unrelated to teardown), but the underlying teardown-leak is a separate issue worth tracking. Skipping per CI gate; auto-merge stays armed.

[HongmingWang-Rabbit]

Skip (no-change re-pass): same E2E Staging SaaS failure from 23:04:31Z, no new commits since. Holding pattern from my prior comment still applies — feature test passes, teardown-verify fails on LEAK: org … still present post-teardown. Auto-merge stays armed; will re-evaluate when the underlying teardown-cascade issue is fixed (separate from this PR's content).

[github-actions]

🔒 Auto-merge disabled — new commit (dccec65) pushed after auto-merge was enabled. The merge queue locks SHAs at entry, so subsequent pushes can race. Verify the new commit and re-enable with gh pr merge --auto.