Skip to content

fix(ci): scope merge-queue runs to the changed subtrees - #574

Merged
balajinvda merged 1 commit into
mainfrom
fix/merge-queue-change-aware
Jul 30, 2026
Merged

fix(ci): scope merge-queue runs to the changed subtrees#574
balajinvda merged 1 commit into
mainfrom
fix/merge-queue-change-aware

Conversation

@balajinvda

@balajinvda balajinvda commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Why

Merge queue throughput is the complaint, and the queue was doing far more work
than it needed to: every entry built all 24 matrix rows regardless of what the
pull request changed.

detect resolves its diff base from github.event.before. That field is
populated on push events but does not exist on merge_group, so queue entries
hit the case statement's default arm with an empty BEFORE_SHA and set
run_all=true. The change-aware scheduling that keeps ordinary pull requests
small was inert for precisely the event that gates merging.

Confirmed from a real queue run's detect log:

selected 24 subtree(s): build-container=[byoo-otel-collector, root, ...]

Measurements

Two recent merge_group runs, both 24 rows:

run wall slowest row started immediately rest waited
30526777859 885s 669s (root) 12 ~600s
30519497553 2132s 643s (root) 12 1553-1857s

The slowest row is ~650s in both. The gap between an 885s run and a 2132s run
is almost entirely rows queued waiting for a runner. Full queue-wait
distribution for the slow run:

3, 4, 4, 4, 7, 11, 19, 21, 22, 22, 37, 46,
1553, 1565, 1568, 1568, 1569, 1569, 1574, 1580, 1851, 1855, 1855, 1857

Twelve rows start at once, then a 26-minute cliff. That shape is runner
starvation, not a max-parallel cap: a binding cap of 8 would produce tiers
roughly one row-duration apart, not a bimodal split at 12.

This change adds no capacity. It removes demand, which is the half we control
from inside the repository.

What changed

A merge_group arm in the case statement using
github.event.merge_group.base_sha. The queue head is the base plus the queued
commits, so diffing from it yields the pull request's own changes, and for a
batched entry every pull request in that batch.

Testing

Validated by simulating a queue-shaped history against the real repository: a
branch at origin/main plus one commit touching a single subtree, run through
the exact case logic.

merge_group with base_sha  -> only the touched path
merge_group, base missing  -> FULL (no base_sha)
merge_group, bogus base    -> FULL (base not in history)
push with empty before     -> FULL (the old behaviour, i.e. the bug)

Both degraded paths fall back to the full matrix rather than selecting nothing,
so a payload change or a shallow checkout cannot silently skip validation.

This pull request touches the workflow itself, so its own runs select the full
matrix by design. The effect shows on the next queue entry that does not touch
workflow files.

Notes

max-parallel is deliberately left alone. The data above shows it is not the
binding constraint, and it exists to stop ~20 runners pulling
actions/checkout at once and tripping GitHub's action-download rate limit
(HTTP 429 in "Set up job"). Removing it risks trading slow for flaky without
addressing the wait.

The remaining constraint is hosted-runner concurrency at the organisation
level, which cannot be changed from this repository.

References

None

Dependencies

None.

Summary by CodeRabbit

  • Bug Fixes
    • Improved change detection for merge queue builds.
    • Ensured affected checks run against the correct set of changes.
    • Added fallback handling to run the full validation suite when change information is unavailable or cannot be calculated.

Every merge-queue entry built all 24 matrix rows regardless of what the pull
request touched.

detect resolves its diff base from `github.event.before`. That field exists on
push events but not on merge_group, so queue entries fell through the case
statement's default arm with an empty BEFORE_SHA and set run_all=true. The
change-aware scheduling that keeps ordinary pull requests small was inert for
exactly the event that gates merging.

merge_group carries the queue base as `github.event.merge_group.base_sha`. The
queue head is the base plus the queued commits, so diffing from it yields the
pull request's own changes, and for a batched entry every pull request in the
batch.

Measured on two recent queue runs, both 24 rows: the slowest row was 669s and
643s, while wall clock was 885s and 2132s. The difference is rows waiting for a
runner, not work. Twelve rows started immediately and the rest waited 26 to 31
minutes, which is runner starvation rather than a max-parallel cap. This does
not add capacity; it removes demand, which is the half we control here.

Both degraded paths stay on the safe side: a missing base_sha, or one absent
from history, falls back to the full matrix rather than selecting nothing.

Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@balajinvda
balajinvda requested a review from a team as a code owner July 30, 2026 15:28
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6da9502c-a044-4773-aa7c-b29ec32faf83

📥 Commits

Reviewing files that changed from the base of the PR and between cbab2ea and 6029f28.

📒 Files selected for processing (1)
  • .github/workflows/bazel.yml

📝 Walkthrough

Walkthrough

The Bazel workflow now handles merge_group events using the event’s base SHA, computes targeted changed-file scheduling when possible, and falls back to the full matrix when required data or diff history is unavailable.

Changes

Bazel merge-group scheduling

Layer / File(s) Summary
Merge-group base and diff detection
.github/workflows/bazel.yml
The detect job reads merge_group.base_sha, validates the base commit, computes the three-dot diff, and falls back to full-matrix execution when validation or diffing fails.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • NVIDIA/nvcf#553: Both changes update Bazel workflow matrix and subtree-selection behavior.

Suggested reviewers: famousdirector

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title follows Conventional Commits and accurately describes the CI workflow bug fix to scope merge-queue runs.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/merge-queue-change-aware

Comment @coderabbitai help to get the list of available commands.

@balajinvda
balajinvda added this pull request to the merge queue Jul 30, 2026
Merged via the queue into main with commit fb6a6c9 Jul 30, 2026
38 checks passed
@balajinvda
balajinvda deleted the fix/merge-queue-change-aware branch July 30, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants