Skip to content

ci(janitor): cancel a queued DEV publish run once a newer publish covered its commit (116 queued, oldest 37 h) - #28976

Merged
teamleaderleo merged 1 commit into
mainfrom
ci/janitor-dev-publish-latest-wins
Oct 31, 2026
Merged

teamleaderleo merged 1 commit into
mainfrom
ci/janitor-dev-publish-latest-wins

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Leo: "idk i feel like the ci wastage chat all got f***ing not much done" and "didnt we have a bunch of dudes doing ci/build optimizations too to reduce waste and redundancy and jam stuff up the wazoo".

What was wasting

At 05:00Z on 2026-10-31, 116 "cmux NEXT DEV daily · publish N" runs were queued on main. The oldest was from 2026-10-29 15:34Z, 37 h earlier. Each one parked a Verify job and a tip DMG job on a Mac label: about 106 on blacksmith-12vcpu-macos-26 (no starts in about a day) and the newer ones on glaeda-std-xcode-26.6, where 4 of 65 runners are online.

The janitor's superseded pass could never skip them, for two reasons:

  • Every publish run runs on main at main's head, so superseded_unstarted() can't tell the runs apart by head.
  • Their Linux Sign job runs within minutes, so build_started() counts every one of them as started.

None of them can deliver. The Publish job's not_newer() skips a source commit that the publish record already contains, and the tip feed refuses a build that is not newer than the one it has. Leo dropped DEV builds as a priority, so this only stops the waste; it revives no DEV work.

Change

delivery_workflows.dev_publish_superseded() adds one more "certain not to deliver" case to the delivery rule. A run qualifies only when all of these hold:

  1. It is a DEV publish run, none of its jobs is running (in progress, or queued with a runner assigned), and its Publish job has not run.
  2. Nothing in it has moved for 20 min (STALE_WAIT).
  3. A later DEV publish run completed in success with its Publish job succeeding.
  4. That run's source commit contains this run's source commit (GitHub compare ahead or identical).

Anything it cannot read is a keep. The janitor's skip_superseded_dev_publish() applies the rule oldest first. It re-reads the run and its jobs right before each cancel, and is capped at 20 cancels and 25 run reads a sweep. Started nightly, DEV and promote builds stay protected: a run with any running job, or whose Publish job has run, is never touched.

Performance

metric before after delta harness/run link
Queued DEV publish runs on main 116 (oldest 37 h) at most the publish runs newer than the last published one (7 at 05:00Z) about -109 live dry run of this head's rule against GitHub at 05:20Z (script below)
Mac jobs parked by those runs about 218 (Verify + tip DMG) about 14 about -204 same
Live dry run, first 36 runs read 0 cancellable (old rule) 29 cancel, 7 keep (the 7 are newer than the last publish) +29 same; the other 80 reads hit the drained shared REST quota and were keeps

The dry run called skip_superseded_dev_publish(..., dry_run=True) with a client that reads through gh api. It cancelled nothing.

Tests

  • tests/test_delivery_workflows_cancel.py (DevPublishSupersededTests, 41 pass): the run 40946510427 shape (tip DMG finished, Verify waiting) is superseded, and a running job, a run Publish job, a recent run, an unpublished or older winner, and an unread or unrelated source are each kept.
  • tests/test_ci_queue_janitor.py (SupersededDevPublishTests, 7 pass, plus the existing SupersededDeliveryTests): the sweep cancels only the older waiting run, keeps a run whose Mac job started between the reads, respects the dry run, and stays within its cap and read budget.

Prior art acknowledged: #26938, #26200, #25882 (the stale rule and its orphan protection, unchanged), #28852 and #27114 (other janitor passes), #26585 and #28431.

Risk: a cancelled run is one that publishes nothing anyway. The worst case is that a newer source fails to publish after its Publish job succeeded, and the next dispatcher tick asks again.

Agent session: agent session

🤖 Generated with Claude Code

…ered its commit

All DEV publish runs run on main at whatever main's head was, and their Linux
sign job runs within minutes, so superseded_unstarted() could never skip one:
115 sat queued on 2026-10-31 (oldest 37 h), each holding a Verify and a tip DMG
job on a Mac label. Such a run can never publish once a newer publish run has
published a source commit containing its own (the publisher's not_newer() and
the tip feed refuse it), so it is certain not to deliver.

dev_publish_superseded() is that rule (nothing running, Publish not run, 20 min
idle, a newer publish run whose Publish job succeeded, and its source contains
this one); the janitor's skip_superseded_dev_publish() applies it, re-reading
before each cancel, capped at 20 cancels and 25 reads a sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d750cf10922c785efa64
@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at f9e462a

SHIP at f9e462a

Visual verdict: Not applicable; CI-only change with no UI diff.

Focused tests pass: 345 tests.

Follow-ups: none

@teamleaderleo
teamleaderleo merged commit 94c8f09 into main Oct 31, 2026
43 checks passed
@teamleaderleo
teamleaderleo deleted the ci/janitor-dev-publish-latest-wins branch October 31, 2026 07:01
@github-actions

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Merge receipt for f9e462a3ac: every check was green at merge (9 verified; 17 skipped by policy). Full suite runs on main after merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant