Repository navigation
ci(janitor): cancel a queued DEV publish run once a newer publish covered its commit (116 queued, oldest 37 h) - #28976
Merged
Conversation
…ered its commit All DEV publish runs run on main at whatever main's head was, and their Linux sign job runs within minutes, so superseded_unstarted() could never skip one: 115 sat queued on 2026-10-31 (oldest 37 h), each holding a Verify and a tip DMG job on a Mac label. Such a run can never publish once a newer publish run has published a source commit containing its own (the publisher's not_newer() and the tip feed refuse it), so it is certain not to deliver. dev_publish_superseded() is that rule (nothing running, Publish not run, 20 min idle, a newer publish run whose Publish job succeeded, and its source contains this one); the janitor's skip_superseded_dev_publish() applies it, re-reading before each cancel, capped at 20 cancels and 25 reads a sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-d750cf10922c785efa64
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
Collaborator
Author
|
Owned by an agent session · session |
Collaborator
Author
Contributor
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Merge receipt for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leo: "idk i feel like the ci wastage chat all got f***ing not much done" and "didnt we have a bunch of dudes doing ci/build optimizations too to reduce waste and redundancy and jam stuff up the wazoo".
What was wasting
At 05:00Z on 2026-10-31, 116 "cmux NEXT DEV daily · publish N" runs were queued on main. The oldest was from 2026-10-29 15:34Z, 37 h earlier. Each one parked a Verify job and a tip DMG job on a Mac label: about 106 on blacksmith-12vcpu-macos-26 (no starts in about a day) and the newer ones on glaeda-std-xcode-26.6, where 4 of 65 runners are online.
The janitor's superseded pass could never skip them, for two reasons:
superseded_unstarted()can't tell the runs apart by head.build_started()counts every one of them as started.None of them can deliver. The Publish job's
not_newer()skips a source commit that the publish record already contains, and the tip feed refuses a build that is not newer than the one it has. Leo dropped DEV builds as a priority, so this only stops the waste; it revives no DEV work.Change
delivery_workflows.dev_publish_superseded()adds one more "certain not to deliver" case to the delivery rule. A run qualifies only when all of these hold:aheadoridentical).Anything it cannot read is a keep. The janitor's
skip_superseded_dev_publish()applies the rule oldest first. It re-reads the run and its jobs right before each cancel, and is capped at 20 cancels and 25 run reads a sweep. Started nightly, DEV and promote builds stay protected: a run with any running job, or whose Publish job has run, is never touched.Performance
The dry run called
skip_superseded_dev_publish(..., dry_run=True)with a client that reads throughgh api. It cancelled nothing.Tests
tests/test_delivery_workflows_cancel.py(DevPublishSupersededTests, 41 pass): the run 40946510427 shape (tip DMG finished, Verify waiting) is superseded, and a running job, a run Publish job, a recent run, an unpublished or older winner, and an unread or unrelated source are each kept.tests/test_ci_queue_janitor.py(SupersededDevPublishTests, 7 pass, plus the existing SupersededDeliveryTests): the sweep cancels only the older waiting run, keeps a run whose Mac job started between the reads, respects the dry run, and stays within its cap and read budget.Prior art acknowledged: #26938, #26200, #25882 (the stale rule and its orphan protection, unchanged), #28852 and #27114 (other janitor passes), #26585 and #28431.
Risk: a cancelled run is one that publishes nothing anyway. The worst case is that a newer source fails to publish after its Publish job succeeded, and the next dispatcher tick asks again.
Agent session: agent session
🤖 Generated with Claude Code