Skip to content

ci(cloud-deploy): per-env deploy-job concurrency so prod deploys can't be cancelled mid-flight (#11640) - #11653

Merged
lalalune merged 2 commits into
developfrom
fix/11640-deploy-job-concurrency
Jul 2, 2026
Merged

ci(cloud-deploy): per-env deploy-job concurrency so prod deploys can't be cancelled mid-flight (#11640)#11653
lalalune merged 2 commits into
developfrom
fix/11640-deploy-job-concurrency

Conversation

@NubsCarson

Copy link
Copy Markdown
Member

Closes #11640. [cloud-money] / deploy lane.

Bug (launch-critical)

migrate-db has a job-level cloud-db-migrate-<env> concurrency group (cancel-in-progress:false), but the deploy jobs don't — they inherited only the workflow-level group, which didn't protect the in-flight job. With rapid develop→main promotes, every newer main run preempted the running Deploy API Worker job → no Worker deploy completed since 19:13Z, so prod ran stale code (only Pages/console updated). Evidence on the issue: Deploy API Worker cancelled on 854455d + 1fb1c53×2.

Fix

Per-job, per-env concurrency (job-level groups are repo-wide → serialize the same job across runs, queue never cancel):

  • deploy-apicloud-cf-deploy-api-<production|staging>, cancel-in-progress:false
  • deploy-console / deploy-app…-<pr-N|production|staging>, cancel-in-progress only for PR previews

Distinct group per job so the three deploys still parallelize within a run; PR previews still dedupe.

Verification

Can't run a workflow locally, so: YAML validated (yaml.safe_load parses; all 4 jobs carry concurrency), groups confirmed distinct per job, and the pattern mirrors the already-working migrate-db protection + the workflow-level env resolution. Prove-out is the next promote completing a Worker deploy.

Touches the multi-agent-contested cloud-cf-deploy.yml — flagging rather than self-merging; resolve any conflict to develop's version. cc @standujar (deploy/ops)

…deploy can't be cancelled mid-flight (#11640)

migrate-db already has a job-level `cloud-db-migrate-<env>` concurrency
group with cancel-in-progress:false, but the deploy-api / deploy-console /
deploy-app jobs had none — they inherited only the workflow-level group,
which did not protect the in-flight job. With the fleet's rapid develop→main
promote cadence, every newer main run preempted the running Worker deploy, so
NO Worker deploy completed since 19:13Z and prod ran stale code while only
Pages/console updated.

Add a per-job, per-env concurrency group (job-level groups are repo-wide, so
they serialize the same job ACROSS runs — queue, never cancel):
  - deploy-api    → cloud-cf-deploy-api-<production|staging>, cancel-in-progress:false
  - deploy-console→ cloud-cf-deploy-console-<pr-N|production|staging>
  - deploy-app    → cloud-cf-deploy-app-<pr-N|production|staging>
PR previews keep a per-PR group that DOES dedupe (cancel-in-progress:true);
main/dispatch-prod and develop/staging never cancel. Distinct group per job so
the three still run in parallel within a run. YAML validated (parses; 4/4 jobs
carry concurrency).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 24dd6878-5970-4381-9228-64e0c5a7219e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/11640-deploy-job-concurrency

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@lalalune

lalalune commented Jul 2, 2026

Copy link
Copy Markdown
Member

Reviewed the workflow-only fix and pushed evidence commit cefa56e9a0.

Validation performed:

python3 YAML parse + job concurrency check
=> migrate-db, deploy-api, deploy-console, deploy-app all have job-level concurrency

migrate-db: cloud-db-migrate-<production|staging>, cancel-in-progress: false
deploy-api: cloud-cf-deploy-api-<production|staging>, cancel-in-progress: false
deploy-console: cloud-cf-deploy-console-<pr-N|production|staging>, PR previews cancel, branch deploys queue
deploy-app: cloud-cf-deploy-app-<pr-N|production|staging>, PR previews cancel, branch deploys queue

git diff --check origin/develop...HEAD
=> clean

I also added .github/issue-evidence/11640-cloud-deploy-job-concurrency.md. Biome ignores that markdown evidence path, so there is no meaningful Biome file check for this PR beyond YAML parse/diff-check.

Manual review: the groups are distinct per deploy job, so API/Console/App can still run in parallel within a single workflow run; job-level groups serialize only the same deploy job across runs. This matches the issue intent and preserves PR preview dedupe for Pages jobs.

@lalalune lalalune left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after workflow review and evidence commit cefa56e9a0. The job-level concurrency groups protect production/staging deploy jobs from mid-flight cancellation while preserving PR preview dedupe and intra-run API/Console/App parallelism.

@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Claude encountered an error —— View job


I'll analyze this and get back to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci/deploy: prod Worker/Console deploy jobs get cancelled by rapid promotes — need cancel-in-progress:false like migrate

3 participants