staging → main: pipeline self-healing fixes (#2203 + #2204) — final manual bridge - #2205
Merged
Conversation
The auto-promote-staging.yml gate-check (line 99) treats "workflow didn't run" as failure. Path-filtered triggers on E2E API Smoke Test and E2E Staging Canvas meant a platform-only or test-only push to staging — say, the prior PR #2201 which only touched tests/e2e/test_staging_full_saas.sh — never triggered the canvas workflow, and auto-promote saw `missing/none`, marked all_green=false, and aborted. Same class for any push that doesn't touch the gate's watched paths. Dead-lock by design, never noticed because the gate was new. Fix per Design B (always-run + fast-skip): - Drop `paths:` from the push/pull_request triggers on both gate workflows. The workflow now always fires on every staging+main push/PR. - Add a `detect-changes` job using `dorny/paths-filter@v3` that decides whether to do real work, scoped to the same paths the trigger filter used to watch. - Real work job (e2e-api / playwright) gates on `needs: detect-changes; if: needs.detect-changes.outputs.X == 'true'`. - Add a sibling `no-op` job that runs when the filter output is false, emitting `::notice::… no-op pass`. The workflow run's conclusion is `success` either way — auto-promote sees green and proceeds. manual `workflow_dispatch` and the weekly canvas `schedule` short- circuit detect-changes to always-run — those triggers exist precisely to exercise the suite and shouldn't be silently no-op'd. Why this approach over making auto-promote-staging smarter: The alternative (Design A, considered + rejected) was to teach auto-promote-staging to read each gate's `paths:` filter and treat "no run because filter excluded the commit" as conditional pass. That couples auto-promote to other workflows' YAML schema and breaks silently if a gate is renamed or its filter changes. Design B keeps the auto-promote contract simple ("each gate emits success") and makes each gate self-describing — adding a new gate doesn't require touching auto-promote. Cost: ~10-30s of runner overhead per gate per push for the no-op when paths don't match. Negligible vs the alternative of dead-locked auto-promote chains. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…esult fix(ci): e2e gates always emit a result so auto-promote can read it
Observed 2026-04-28: auto-promote ran for staging head 96955f7 with all gates actually green (verified via /commits/<sha>/check-runs API) yet `check-all-gates-green` reported `CodeQL → missing/none` and aborted. Same SHA was promotable; auto-promote couldn't see it. Cause: `gh run list --workflow="CodeQL"` matched two workflows in this repo: - codeql.yml (explicit, scans both staging and main) - codeql (GitHub UI-configured Code-quality default setup, internal, scans default branch only) gh CLI rejects ambiguous `--workflow=<name>` lookups and returns no result → the gate fell through to `missing/none` and ALL_GREEN was set false. Every staging push since both names existed has been silently dead-locked. Fix: switch GATES from display-name strings to workflow file paths. File paths are the unique identifier for a workflow file in .github/workflows/; display names are decoration and can collide. The same `gh run list --workflow=<file.yml>` query that fails on "CodeQL" succeeds on "codeql.yml" because the file path resolves unambiguously. No behavior change for the other three gates (CI, E2E Canvas, E2E API Smoke) since their names didn't collide — they keep working, they just identify by ci.yml / e2e-staging-canvas.yml / e2e-api.yml now. The log line shape changes from `CI → completed/success` to `ci.yml → completed/success` which is fine for ops grep. When adding/removing a gate going forward: file paths only. Keep branch-protection required-checks (check-run display names) in sync as a separate manual step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…file-paths fix(ci): auto-promote gate-check uses workflow file paths, not names
HongmingWang-Rabbit
requested a review
from hongmingwang-moleculeai
as a code owner
April 28, 2026 20:24
3 tasks
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…2205) The `E2E API Smoke Test` REQUIRED gate (and the sibling local-platform E2E workflows) started the platform in the background and waited for /health with a fixed 30×1s loop (~30s). The platform binds /health only AFTER applying the FULL migration chain on cold start; that chain now reaches past the 30s window (the run log gets to 20260523000000_schedule_consecutive_sdk_errors.up.sql before "Platform starting on :PORT"), so the health loop expired before the server was reachable → downstream E2E never ran → main went red. A fixed budget is brittle by construction because the migration chain grows every release. Fix (deterministic, not a bigger magic number): - Poll /health on a generous, clearly-commented wall-clock budget (180s) that comfortably exceeds cold-start + full-migration time and is robust to the chain continuing to grow. /health returning 200 is the real readiness signal (migrations done + server listening). - Still fail fast + loud on a genuinely dead platform: if the backgrounded platform-server PID has exited (e.g. a broken migration crashed it), stop immediately and dump the platform log — we never mask a real startup failure, and we never wait out the full budget for a process that is already gone. - On true timeout, dump the platform log tail and fail with ::error::. Applied identically to the four workflows sharing the 30×1s platform-/health pattern: e2e-api, e2e-chat, e2e-peer-visibility, e2e-legacy-advisory. The unrelated Postgres-readiness `seq 1 30` waits (which are not gated on the migration chain) are intentionally left unchanged. curl usage avoids the -w '%{http_code}' status-capture shape, so lint-curl-status-capture passes; lint-workflow-yaml passes on all 56 files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…solves #2205) Issue #2205 reports E2E API Smoke health-wait times out while platform migrations are still running. The previous step polled /health for 30s with no migration awareness, so it could exit 0 before the DB was actually usable, causing downstream steps to flake on "no such table". Hybrid fix: 1. Bump probe count 30→300 (1s sleep each, 5min ceiling — enough for the full migration chain on cold-cache runners). 2. Gate exit on the same workspaces-table existence check the downstream "Assert migrations applied" step uses. We now only declare /health success when both /health=200 AND the workspaces table is present. 3. The downstream "Assert migrations applied" step stays as a defense-in-depth final check; with the new gate it should always pass on a clean run. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
One-time manual bridge so the auto-promote-staging fixes themselves reach `main`. After this lands, the chain is self-healing:
After both are on main, every staging push that gets all 4 gates green will auto-fast-forward main with no human intervention.
Why manual
The bug we just fixed in #2204 lives on `main` until this bridge merges. Auto-promote-staging.yml on main still uses the buggy display-name lookup, sees CodeQL → missing/none, refuses to fast-forward. Bypassing it once via direct API merge gets `main` to a state where future runs use the fixed code.
What's being promoted
After this lands
🤖 Generated with Claude Code