Skip to content

ci(janitor): carry the gates' own state per glaeda label in macos-pool-load - #28312

Merged
teamleaderleo merged 1 commit into
mainfrom
dep-fleet-2/janitor-fleet-gates
Oct 30, 2026
Merged

teamleaderleo merged 1 commit into
mainfrom
dep-fleet-2/janitor-fleet-gates

Conversation

@teamleaderleo

Copy link
Copy Markdown
Collaborator

Why

GitHub lists a runner whose glaeda gate holds its listener off as offline, the same as a dead one. The overflow route reads GitHub's runner list, so it cannot tell "this label is full" from "this label is broken", and the two disagree with the gates. The fleet dashboard now keeps the gates' own state in the CI_FLEET_GATES repository variable (manaflow-ai/cmuxterm-hq#5321). This PR puts that state in the macos-pool-load artifact the pickers already download.

What

  • The janitor's sweep step gets FLEET_GATES: ${{ vars.CI_FLEET_GATES }} through env only. A variable read costs no API request, and the value is never expanded into a script.
  • fleet_gates() in scripts/ci/queue_janitor.py checks the reading. Any field out of shape refuses the whole reading, which is the safe side because readers treat missing as full. It keeps only the documented fields and writes them into the snapshot as gates. With no reading it writes gates_error instead.
  • Per glaeda label: listening, busy, held_offline {total, units, cpu, livelock, other}, down, unreported, queued, oldest_queued_s, as_of, plus a top-level as_of. The meaning of each field and the staleness rule are documented beside fleet_gates() and in the workflow header. A reading over 5 min old is stale, and a reader treats stale or missing as full for jobs queued over 3 min. Readers judge as_of against their own clock, never the snapshot's generated_at.

The readers' use of gates (the pickers' "label is full" decision) is a separate change, owned by the overflow-route lane.

Tests

FleetGatesTests in tests/test_ci_queue_janitor.py has 5 tests: the fields kept per label, refused shapes (bad JSON, version, as_of, negative or bool counts, a missing reason, a non-glaeda label), a sweep writing gates, a sweep writing gates_error, and the workflow passing the variable through env only. All 5 fail on main (red) and pass here (green). The full test_ci_queue_janitor.py suite passes, and so do the other tests that read the janitor (change areas, fork routing, health report, owned-pool rescue, warm state, pr_runner_pool).

Risk

The change is additive: one new snapshot key. pr_runner_pool.py and the other readers ignore keys they do not know.

Agent session: agent session

🤖 Generated with Claude Code

…l-load

GitHub lists a runner whose glaeda gate holds its listener off as offline, the same as a dead
one, so the overflow route cannot tell a full label from a broken one. The fleet dashboard now
keeps the gates' own state in the CI_FLEET_GATES repository variable. The janitor reads it
through env (no API request), checks it against the documented fields, and writes it into the
macos-pool-load snapshot as `gates`, or says why in `gates_error`. The fields and the readers'
staleness rule (over 5 min is stale; stale or missing is full for jobs queued over 3 min) are
documented beside fleet_gates().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@teamleaderleo
teamleaderleo merged commit 4ae385f into main Oct 30, 2026
37 checks passed
@teamleaderleo
teamleaderleo deleted the dep-fleet-2/janitor-fleet-gates branch October 30, 2026 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant