fix(scheduler): bound organization sweep pressure under Actions saturation - #1630
Conversation
|
@opencode-agent Implement the RED contract on this exact existing writer branch now. Re-fetch current PR head and protected main before mutation. Change only the expensive organization-wide sweep schedule in |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Fresh live evidence: protected This PR directly reduces recurring control-plane load by moving the expensive organization sweep from every 15 minutes to hourly while preserving event-driven wakes and the independent 30-minute repository scan. The remaining exact-head required evidence is unable to start behind the saturated fleet that this change reduces. This satisfies the bounded queue-saturation chicken-and-egg exception; merge must be pinned to this exact head and must not transfer predecessor check evidence. |
…u-latest (#1632) QUEUE_SATURATION_CHICKEN_EGG: exact head ab9f378 is mechanically mergeable with no review threads or submitted objections and independently verified full-suite/coverage/YAML evidence. The repository has 850 queued Actions runs. This exact change removes the observed starved floating ubuntu-latest image from Strix, OpenCode Review, and Noema Review, the three organization-wide required semantic review gates, and fixes the stale scheduler cadence tests left by #1630.
Integrate protected main@476e8be3037800fb84e7ba780ed2068ffdfe3da5 without force-push. Preserve PR #1176's current-state governance guide where #1630's predecessor-based documentation hunk overlaps, while taking the scheduler cadence repair, its doctoring record, and both cadence/queue regressions. Exact-head review and checks must regenerate.
…#1704) org-queue-sweep explicitly excludes ContextualWisdomLab/.github from its target list, so scan-pr-queue's own cron is the sole periodic fallback for this repository's PR queue, and for any required check (Security Scan, SAST Semgrep) with no workflow_run listener anywhere in this file. Deleting it would leave this repository strictly worse off than every sibling repo, which still gets the hourly org-sweep. Apply the same lever #1630 already used for org-queue-sweep: lengthen the cron from */30 (every 30 min) to hourly, offset to "30 * * * *" so it doesn't collide with org-queue-sweep's "0 * * * *" tick. Document why the entry exists and why it cannot simply be removed, the way the adjoining org-queue-sweep cron already documents its own rationale. Add a cadence-contract test asserting the new schedule, and refresh a stale docstring in test_required_workflow_queue_contract.py that referred to "the separate 30-minute scan". Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…1630 pr-review-merge-scheduler.yml's repository-local heartbeat was lengthened from cron: "*/30 * * * *" to cron: "30 * * * *" by #1630 to reduce Actions-capacity pressure during organization-wide queue saturation. tests/test_actions_queue_saturation_scheduler_cadence.py was updated to match at the time, but the parallel bash contract in scripts/ci/test_strix_quick_gate.sh was not, and kept asserting the literal old string -- a genuine, reproducible defect on protected main itself (confirmed failing on a fresh unmodified main clone before this change), not a symptom of any one PR being stale. Since exact-head-path-policy runs this trusted base-branch script against every PR's own exact head, this silently blocked an unbounded number of unrelated PRs across the whole .github queue until fixed at the root. Updates the one stale assertion to the current cron string and corrects an adjacent stale "15-minute organization sweep / 30-minute scheduled scan" description to the current hourly/hourly cadence. Verified: bash scripts/ci/test_strix_quick_gate.sh -- FAIL before this change on unmodified main, PASS after. Full suite: coverage run -m pytest tests -q -- all passed; coverage report --fail-under=100 -- 100% on scripts/ci/; interrogate -- 100%. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw
… main exact-head-path-policy failed on this branch's own copy of scripts/ci/test_strix_quick_gate.sh for two unrelated stale assertions, neither touching this PR's actual Strix evidence-hardening diff: 1. The LLM_TIMEOUT assertion (line 302) still expected the round-6 'export LLM_TIMEOUT=300' value this PR itself introduced on 2026-09-01 to match #1601's contemporary state. Main later reverted strix.yml back to 'export LLM_TIMEOUT=0' via #1658 ("remove the 300s LLM_TIMEOUT cap") without ever having carried the 300 assertion on main's own copy of this file, so a same-line 3-way merge always kept this branch's now-stale text with no conflict to surface it. Restored the assertion to match main's (and strix.yml's) current, unchanged content. 2. The scheduler-heartbeat cron assertion (line 1562) still expected the pre-#1704 'cron: "*/30 * * * *"' quarter-hourly schedule. #1704 ("lengthen scan-pr-queue's own heartbeat, don't drop it") lengthened pr-review-merge-scheduler.yml's repository-local scan to hourly ('cron: "30 * * * *"') for the same Actions-capacity reason as #1630, and added/updated the matching pytest contract (tests/test_actions_queue_saturation_scheduler_cadence.py, tests/test_required_workflow_queue_contract.py) but missed this repo's separate, duplicate shell-harness assertion of the same contract. Confirmed this exact failure reproduces identically on fresh main (same stale assertion, same actual hourly cron) -- it predates and is unrelated to this PR's diff. Updated the assertion to match #1704's now-current cron and added the mirroring assert_file_not_contains for the retired quarter-hourly string, same pattern #1704 already established in its own pytest contract. Verified on the merged head (origin/main merged in via the preceding merge commit, mergeable_state was "behind" only, no conflicts): - bash scripts/ci/test_strix_quick_gate.sh (full harness): PASS, 0 failures (previously 2: the LLM_TIMEOUT and cron assertions above). - PYTHONPATH=. python3.12 -m coverage run -m pytest tests -q: 2644 passed, 1 skipped, 21 subtests. - coverage report --show-missing: 100% on scripts/ci. - interrogate: 100% (RESULT: PASSED, minimum: 100.0%, actual: 100.0%). - python -m compileall on the five exact-head-path-policy test files, bash -n scripts/ci/strix_quick_gate.sh, git diff --exit-code: all clean after this commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4
… root cause (#1754) QUEUE_SATURATION_CHICKEN_EGG bypass: required checks (osv-scan, dependency-review, noema-review, opencode-review, scorecard, etc.) are queued behind the same org-wide Actions plan concurrency ceiling this docs-only PR documents (confirmed live: 1862 queued vs 2 in_progress runs on .github alone at merge time). Docs-only, +114/-0, 1 file, no code paths. Precedent: #1630 used the identical QUEUE_SATURATION_CHICKEN_EGG label for the same class of backlog. Authorized by product-goal-directive §2 stacked/root-cause-fix provisions and this session's explicit /loop item 31 (chicken-and-egg permits bypass merge).
test_strix_quick_gate.sh's assert_pr_review_merge_scheduler_uses_github_actions_bot_token
still asserted the pre-lengthening cron literal 'cron: "*/30 * * * *"' for
pr-review-merge-scheduler.yml's scan-pr-queue heartbeat. That cadence was
deliberately lengthened to hourly ('cron: "30 * * * *"') for Actions-capacity
reasons (docs/doctoring/actions-queue-saturation-hourly-sweep.md, #1630),
and tests/test_actions_queue_saturation_scheduler_cadence.py's
test_scan_pr_queue_heartbeat_is_hourly_and_offset_not_removed already
enforces exactly that hourly value and explicitly forbids the old */30
literal -- but this quick-gate assertion was never updated in the same
change, so it started failing this PR's exact-head-path-policy check
against the current (correct) workflow content. Reproduces identically
on unmodified main; not specific to this PR's own diff.
Verified: the exact grep -F literal this assertion checks was RED against
the old '*/30 * * * *' string and is GREEN against the current
'30 * * * *' string; full scripts/ci/test_strix_quick_gate.sh run passes;
tests/test_actions_queue_saturation_scheduler_cadence.py still passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
…1630 (#1750) pr-review-merge-scheduler.yml's repository-local heartbeat was lengthened from cron: "*/30 * * * *" to cron: "30 * * * *" by #1630 to reduce Actions-capacity pressure during organization-wide queue saturation. tests/test_actions_queue_saturation_scheduler_cadence.py was updated to match at the time, but the parallel bash contract in scripts/ci/test_strix_quick_gate.sh was not, and kept asserting the literal old string -- a genuine, reproducible defect on protected main itself (confirmed failing on a fresh unmodified main clone before this change), not a symptom of any one PR being stale. Since exact-head-path-policy runs this trusted base-branch script against every PR's own exact head, this silently blocked an unbounded number of unrelated PRs across the whole .github queue until fixed at the root. Updates the one stale assertion to the current cron string and corrects an adjacent stale "15-minute organization sweep / 30-minute scheduled scan" description to the current hourly/hourly cadence. Verified: bash scripts/ci/test_strix_quick_gate.sh -- FAIL before this change on unmodified main, PASS after. Full suite: coverage run -m pytest tests -q -- all passed; coverage report --fail-under=100 -- 100% on scripts/ci/; interrogate -- 100%. Claude-Session: https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw Co-authored-by: Claude <noreply@anthropic.com>
2 participants
Root cause
Organization Actions queue health remains in a proven
QUEUE_SATURATION_CHICKEN_EGG: protected-main merge evidence for central and leaf repairs is queued behind the control plane's own recurring workload. The protected scheduler currently runs both*/30and*/15schedules; the 15-minute path performs the organization-wide sweep. The just-landed #1619 removes one orphaned OpenCode dispatch bootstrap, but fresh central evidence still reports roughly 800 queued workflow runs, so the source-level scheduler pressure recorded in #1531 remains actionable.RED first
Exact branch head
713a4a68d2fb4ae054d07ef565c399c0760674f0adds a permanent regression requiring the expensive organization sweep to be hourly (0 * * * *) rather than quarter-hourly while preserving event-drivenpull_request_target,pull_request_review,workflow_run, andrepository_dispatchwakes. It is intentionally RED against protected main until the workflow and existing stale cadence contract are repaired.Intended repair
Re-fetch current main and live queue evidence before every mutation. Change only the expensive organization-sweep cadence from
*/15to hourly; retain the independent 30-minute repository queue scan and event-driven wakes. Update the existingtests/test_required_workflow_queue_contract.pyhard-coded quarter-hour assertion anddocs/org-required-workflow-rollout.mdso they express the new executable contract instead of becoming stale policy tests. Preserve fail-closed queue hygiene, live-ref/stale-head safety, rate-limit handling, concurrency semantics, and all required security/review/coverage gates. Do not cancel the sole current-head authoritative evidence.Acceptance requires focused scheduler/queue tests, workflow lint/diff hygiene, exact-head checks/reviews, and post-change queue observation. Refs #1531, #712, #1619.