Repository navigation
ci(janitor): carry the gates' own state per glaeda label in macos-pool-load - #28312
Merged
Merged
Conversation
…l-load GitHub lists a runner whose glaeda gate holds its listener off as offline, the same as a dead one, so the overflow route cannot tell a full label from a broken one. The fleet dashboard now keeps the gates' own state in the CI_FLEET_GATES repository variable. The janitor reads it through env (no API request), checks it against the documented fields, and writes it into the macos-pool-load snapshot as `gates`, or says why in `gates_error`. The fields and the readers' staleness rule (over 5 min is stale; stale or missing is full for jobs queued over 3 min) are documented beside fleet_gates(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Owned by an agent session · session |
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
GitHub lists a runner whose glaeda gate holds its listener off as offline, the same as a dead one. The overflow route reads GitHub's runner list, so it cannot tell "this label is full" from "this label is broken", and the two disagree with the gates. The fleet dashboard now keeps the gates' own state in the
CI_FLEET_GATESrepository variable (manaflow-ai/cmuxterm-hq#5321). This PR puts that state in themacos-pool-loadartifact the pickers already download.What
FLEET_GATES: ${{ vars.CI_FLEET_GATES }}through env only. A variable read costs no API request, and the value is never expanded into a script.fleet_gates()in scripts/ci/queue_janitor.py checks the reading. Any field out of shape refuses the whole reading, which is the safe side because readers treat missing as full. It keeps only the documented fields and writes them into the snapshot asgates. With no reading it writesgates_errorinstead.listening,busy,held_offline{total,units,cpu,livelock,other},down,unreported,queued,oldest_queued_s,as_of, plus a top-levelas_of. The meaning of each field and the staleness rule are documented besidefleet_gates()and in the workflow header. A reading over 5 min old is stale, and a reader treats stale or missing as full for jobs queued over 3 min. Readers judgeas_ofagainst their own clock, never the snapshot'sgenerated_at.The readers' use of
gates(the pickers' "label is full" decision) is a separate change, owned by the overflow-route lane.Tests
FleetGatesTestsin tests/test_ci_queue_janitor.py has 5 tests: the fields kept per label, refused shapes (bad JSON, version,as_of, negative or bool counts, a missing reason, a non-glaeda label), a sweep writinggates, a sweep writinggates_error, and the workflow passing the variable through env only. All 5 fail on main (red) and pass here (green). The full test_ci_queue_janitor.py suite passes, and so do the other tests that read the janitor (change areas, fork routing, health report, owned-pool rescue, warm state, pr_runner_pool).Risk
The change is additive: one new snapshot key. pr_runner_pool.py and the other readers ignore keys they do not know.
Agent session: agent session
🤖 Generated with Claude Code