fix(ci): skip e2e/functional tests in merge queue for irrelevant paths by ralphbean · Pull Request #2617 · fullsend-ai/fullsend

ralphbean · 2026-06-24T16:35:11Z

Summary

The merge_group trigger on e2e.yml and functional-tests.yml had no path filter, and the relevance-check step only ran for pull_request_target events
Every PR entering the merge queue ran the full e2e and functional test suites regardless of what files changed (e.g. fix(ci): lint individual commits on PRs to catch invalid prefixes early #2514 which only touched lint.yml)
Extend the relevance check to also run on merge_group events, using the compare API with the merge group base/head SHAs to determine changed files

Test plan

A PR that only touches non-relevant files (e.g. docs, lint config) should skip e2e/functional tests in the merge queue
A PR that touches relevant files (e.g. *.go, eval/**) should still run tests in the merge queue
If the compare API call fails, tests run as a precaution (fail-open)

🤖 Generated with Claude Code

…levant The merge_group trigger had no path filter and the relevance-check step only ran for pull_request_target events, so every PR entering the merge queue ran the full e2e and functional test suites regardless of what files changed. Extend the relevance check to also run on merge_group events, using the compare API with the merge group base/head SHAs to determine changed files. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

qodo-code-review · 2026-06-24T16:36:11Z

PR Summary by Qodo

CI: Skip e2e/functional tests in merge queue for irrelevant changes
🐞 Bug fix ⚙️ Configuration changes 🕐 20-40 Minutes

Description

• Skip merge-queue e2e/functional runs when changed paths are not relevant.
• On merge_group events, detect changed files via GitHub compare API (base/head SHAs).
• Fail open: if change detection fails, run tests to avoid false skips.

Diagram

graph TD A["GitHub event (PR / merge queue)"] --> B["Workflow (e2e / functional)"] --> C["Relevance check step"] --> D["GitHub API (PR files / compare)"] --> E{{"Relevant paths?"}} 
E -->|"yes / unknown"| F["Run test suite"]
E -->|"no"| G["Skip test suite"]

High-Level Assessment

The following are alternative approaches to this PR:

1. Use git diff instead of GitHub API

➕ Avoids API rate limits and reliance on GitHub compare/pulls endpoints
➕ Works offline once refs are available
➖ Requires careful checkout/fetch of the correct base/head SHAs in merge_group context
➖ More git plumbing and higher risk of edge cases (shallow clones, missing refs)

2. Adopt a dedicated paths filter action (e.g., dorny/paths-filter)

➕ Well-known implementation; reduces custom shell and regex plumbing
➕ Potentially clearer configuration for reviewers/maintainers
➖ Adds third-party action dependency and supply-chain surface area
➖ Still needs merge_group-compatible change detection inputs

Recommendation: The current approach is a good fit: it minimally extends existing relevance gating to merge_group, uses authoritative base/head SHAs with the compare API, and fails open to avoid incorrectly skipping test coverage. Consider a git-diff-based approach only if API reliability/rate limits become a recurring issue.

Files changed (2) +34 / -12

Other (2) +34 / -12

e2e.ymlGate merge_group e2e runs using compare-based changed-file detection +17/-6

Gate merge_group e2e runs using compare-based changed-file detection

• Extends the existing 'Check for e2e-relevant changes' step to also run on merge_group events. For merge_group, it queries changed files via the GitHub compare API using base/head SHAs; on failure it marks changes as relevant to run tests.

.github/workflows/e2e.yml

functional-tests.ymlGate merge_group functional tests using compare-based changed-file detection +17/-6

Gate merge_group functional tests using compare-based changed-file detection
• Extends the functional-test relevance check to run on merge_group events. Uses the GitHub compare API to compute changed files for merge queue entries, and fails open (run tests) when file listing fails.
.github/workflows/functional-tests.yml

github-actions · 2026-06-24T16:36:41Z

Site preview

Preview: https://b481b874-site.fullsend-ai.workers.dev

Commit: 885e29556667def5f4659d91a80a569caf061be8

fullsend-ai-review · 2026-06-24T16:38:44Z

🤖 Finished Review · ✅ Success · Started 4:38 PM UTC · Completed 4:48 PM UTC
Commit: 45d230d · View workflow run →

codecov · 2026-06-24T16:38:48Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

qodo-code-review · 2026-06-24T16:39:17Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 51 rules

✅ Skills: writing-user-docs, writing-adrs

1. ~~E2E compare truncation~~ ✓ Resolved 🐞 Bug ☼ Reliability

Description

On merge_group runs, the e2e and functional-tests workflows derive FILES solely from a single `gh
api .../compare/...` response and then skip tests when grep finds no matches. If that compare
response omits some changed files, the workflows can emit relevant=false and skip e2e/functional
tests even when relevant changes are present.

Code

.github/workflows/e2e.yml[R96-101]

+          if [ "$EVENT_NAME" = "merge_group" ]; then
+            FILES=$(gh api "repos/${REPO}/compare/${MERGE_GROUP_BASE}...${MERGE_GROUP_HEAD}" --jq '.files[].filename') || {
+              echo "::warning::Failed to fetch merge group files — running e2e tests as a precaution"
+              echo "relevant=true" >> "$GITHUB_OUTPUT"
+              exit 0
+            }

Relevance

⭐⭐⭐ High
Team previously accepted pagination to avoid truncated API results (PR #225) and fail-open guards
around gh api file lists (PR #2398).
PR-#225
PR-#2398
PR-#2288

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

In the merge_group branch, the workflows use the compare API’s returned filenames as the only
input to the relevance/skip decision, whereas the pull_request_target/PR branch explicitly uses a
paginated file-listing approach (--paginate). Because the merge_group path does not paginate or
otherwise ensure completeness, any omission/truncation in the single compare response would directly
cause the grep/regex check to miss relevant paths, incorrectly set relevant=false, and skip the
corresponding tests.

.github/workflows/e2e.yml[84-114]
.github/workflows/functional-tests.yml[76-106]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
In the `merge_group` relevance checks, the workflows enumerate changed files from a single GitHub `compare` API response and then use that list as the sole basis for deciding whether to skip running tests. If the API response omits files (e.g., response limits/truncation), the grep/regex can miss relevant paths and incorrectly set `relevant=false`, causing e2e and/or functional tests to be skipped despite relevant changes.

## Issue Context
The `pull_request_target` path already uses a paginated endpoint (`--paginate`) to avoid silently missing changed files, but the newer `merge_group` path does not. For merge queue safety, the `merge_group` path should enumerate changed files in a way that cannot silently omit files, or it should detect potential incompleteness and fail open (treat as relevant so tests run).

## Fix Focus Areas
- .github/workflows/e2e.yml[84-114]
- .github/workflows/functional-tests.yml[76-106]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

fullsend-ai-review · 2026-06-24T16:48:40Z

Review

Findings

High

[protected-path] .github/workflows/e2e.yml, .github/workflows/functional-tests.yml — Both files are under the .github/ protected path. This PR has no linked issue to justify the changes to governance/infrastructure files. Human approval is always required for protected-path changes; without a linked issue the justification context is insufficient.
Remediation: Create a GitHub issue documenting the motivation for this change and link it to the PR, or obtain explicit human approval.

The prior review's medium-severity edge-case findings (compare API truncation at 300 files) have been resolved — the new commit adds a FILE_COUNT -ge 300 guard that correctly fails open when truncation is detected.

Previous run

Review

Findings

High

[protected-path] .github/workflows/e2e.yml, .github/workflows/functional-tests.yml — Both files are under the .github/ protected path. This PR has no linked issue to justify the changes to governance/infrastructure files. Human approval is always required for protected-path changes; without a linked issue the justification context is insufficient.
Remediation: Create a GitHub issue documenting the motivation for this change and link it to the PR, or obtain explicit human approval.

Medium

[edge-case] .github/workflows/e2e.yml:96 — The GitHub compare API (/repos/{owner}/{repo}/compare/{base}...{head}) returns at most 300 files in the files array and silently truncates beyond that. Unlike the pulls/files endpoint (which supports --paginate), the compare endpoint has no pagination. If a merge group accumulates more than 300 changed files, the relevance grep could incorrectly determine no relevant files changed, causing tests to be silently skipped. The existing pull_request_target branch handles large file lists via --paginate, but the new merge_group branch has no equivalent safeguard.
Remediation: After fetching the compare response, check if the returned file count equals 300 (the API cap). If so, assume truncation and set relevant=true as a precaution, matching the fail-open pattern already used for API errors.
[edge-case] .github/workflows/functional-tests.yml:88 — Same compare API truncation issue as e2e.yml. The merge group branch uses the compare API which caps the files array at 300 entries without pagination support, potentially causing silent test skipping for large merge groups.
Remediation: Same fix as e2e.yml: check if the returned file count equals the 300-file API cap and default to relevant=true if so.

Labels: PR modifies CI workflow files for e2e and functional test pipelines.

Previous run

Review

Findings

High

[protected-path] .github/workflows/e2e.yml, .github/workflows/functional-tests.yml — Both files are under the .github/ protected path. This PR has no linked issue to justify the changes to governance/infrastructure files. Human approval is always required for protected-path changes; without a linked issue the justification context is insufficient.
Remediation: Create a GitHub issue documenting the motivation for this change and link it to the PR, or obtain explicit human approval.

Medium

[edge-case] .github/workflows/e2e.yml:96 — The GitHub compare API (/repos/{owner}/{repo}/compare/{base}...{head}) returns at most 300 files in the files array and silently truncates beyond that. Unlike the pulls/files endpoint (which supports --paginate), the compare endpoint has no pagination. If a merge group accumulates more than 300 changed files, the relevance grep could incorrectly determine no relevant files changed, causing tests to be silently skipped. The existing pull_request_target branch handles large file lists via --paginate, but the new merge_group branch has no equivalent safeguard.
Remediation: After fetching the compare response, check if the returned file count equals 300 (the API cap). If so, assume truncation and set relevant=true as a precaution, matching the fail-open pattern already used for API errors.
[edge-case] .github/workflows/functional-tests.yml:88 — Same compare API truncation issue as e2e.yml. The merge group branch uses the compare API which caps the files array at 300 entries without pagination support, potentially causing silent test skipping for large merge groups.
Remediation: Same fix as e2e.yml: check if the returned file count equals the 300-file API cap and default to relevant=true if so.

Previous run (2)

Review

Findings

High

[protected-path] .github/workflows/e2e.yml, .github/workflows/functional-tests.yml — Both files are under the .github/ protected path. This PR has no linked issue to justify the changes to governance/infrastructure files. Human approval is always required for protected-path changes; without a linked issue the justification context is insufficient.
Remediation: Create a GitHub issue documenting the motivation for this change and link it to the PR, or obtain explicit human approval.

Medium

[edge-case] .github/workflows/e2e.yml:96 — The GitHub compare API (/repos/{owner}/{repo}/compare/{base}...{head}) returns at most 300 files in the files array and silently truncates beyond that. Unlike the pulls/files endpoint (which supports --paginate), the compare endpoint has no pagination. If a merge group accumulates more than 300 changed files, the relevance grep could incorrectly determine no relevant files changed, causing tests to be silently skipped. The existing pull_request_target branch handles large file lists via --paginate, but the new merge_group branch has no equivalent safeguard.
Remediation: After fetching the compare response, check if the returned file count equals 300 (the API cap). If so, assume truncation and set relevant=true as a precaution, matching the fail-open pattern already used for API errors.
[edge-case] .github/workflows/functional-tests.yml:88 — Same compare API truncation issue as e2e.yml. The merge group branch uses the compare API which caps the files array at 300 entries without pagination support, potentially causing silent test skipping for large merge groups.
Remediation: Same fix as e2e.yml: check if the returned file count equals the 300-file API cap and default to relevant=true if so.

Labels: PR modifies CI workflow files for e2e and functional test pipelines.

fullsend-ai-review

See the review comment for full details.

fullsend-ai-review · 2026-06-24T16:48:42Z

-            echo "relevant=true" >> "$GITHUB_OUTPUT"
-            exit 0
-          }
+          if [ "$EVENT_NAME" = "merge_group" ]; then


[medium] edge-case

The GitHub compare API returns at most 300 files and silently truncates beyond that. Unlike the pulls/files endpoint (which supports --paginate), the compare endpoint has no pagination. If a merge group accumulates more than 300 changed files, the relevance grep could incorrectly skip tests.

Suggested fix: After fetching the compare response, check if the returned file count equals 300 (the API cap). If so, assume truncation and set relevant=true as a precaution.

Good catch. The compare API does cap at 300 files with no pagination. Added a fail-open guard in 885e295 — if we get >= 300 files back, we assume truncation and run tests.

fullsend-ai-review · 2026-06-24T16:48:42Z

-            echo "relevant=true" >> "$GITHUB_OUTPUT"
-            exit 0
-          }
+          if [ "$EVENT_NAME" = "merge_group" ]; then


[medium] edge-case

Same compare API truncation issue as e2e.yml. The merge group branch has no safeguard against the 300-file limit, potentially causing silent test skipping for large merge groups.

Suggested fix: Same fix as e2e.yml: check if the returned file count equals the 300-file API cap and default to relevant=true if so.

Same fix applied here in 885e295.

waynesun09 · 2026-06-24T18:14:39Z

/fs-review

fullsend-ai-review · 2026-06-24T18:17:39Z

🤖 Finished Review · ✅ Success · Started 6:17 PM UTC · Completed 6:28 PM UTC
Commit: 83c41d4 · View workflow run →

fullsend-ai-review

See the review comment for full details.

fullsend-ai-review · 2026-06-24T18:28:26Z

-            echo "relevant=true" >> "$GITHUB_OUTPUT"
-            exit 0
-          }
+          if [ "$EVENT_NAME" = "merge_group" ]; then


[medium] edge-case

The GitHub compare API (/repos/{owner}/{repo}/compare/{base}...{head}) returns at most 300 files in the files array and silently truncates beyond that. Unlike the pulls/files endpoint (which supports --paginate), the compare endpoint has no pagination. If a merge group accumulates more than 300 changed files, the relevance grep could incorrectly determine no relevant files changed, causing tests to be silently skipped.

Suggested fix: After fetching the compare response, check if the returned file count equals 300 (the API cap). If so, assume truncation and set relevant=true as a precaution, matching the fail-open pattern already used for API errors.

fullsend-ai-review · 2026-06-24T18:28:26Z

-            echo "relevant=true" >> "$GITHUB_OUTPUT"
-            exit 0
-          }
+          if [ "$EVENT_NAME" = "merge_group" ]; then


[medium] edge-case

Same compare API truncation issue as e2e.yml. The merge group branch uses the compare API which caps the files array at 300 entries without pagination support, potentially causing silent test skipping for large merge groups.

Suggested fix: Same fix as e2e.yml: check if the returned file count equals the 300-file API cap and default to relevant=true if so.

The GitHub compare API caps at 300 files with no pagination support. Add a check after fetching merge group files: if the count is >= 300, assume possible truncation and run tests as a precaution, matching the existing fail-open pattern for API errors. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

fullsend-ai-review · 2026-06-24T20:53:10Z

🤖 Finished Review · ✅ Success · Started 8:53 PM UTC · Completed 9:04 PM UTC
Commit: 885e295 · View workflow run →

fullsend-ai-review

See the review comment for full details.

fullsend-ai-retro · 2026-06-25T14:03:15Z

🤖 Finished Retro · ✅ Success · Started 2:03 PM UTC · Completed 2:09 PM UTC
Commit: 885e295 · View workflow run →

fullsend-ai-retro · 2026-06-25T14:09:05Z

Retro: PR #2617 — CI merge queue path filter fix

What happened

A human-authored PR by ralphbean added merge queue support to the e2e and functional test workflow path filters (2 files, +34/-12). The review agent ran 3 times:

First review (16:48 UTC) — Caught a real edge-case bug: the GitHub Compare API silently truncates at 300 files, which could cause the relevance check to incorrectly skip tests. Also flagged a high-severity protected-path finding (no linked issue for .github/ changes). Verdict: CHANGES_REQUESTED.
Second review (18:28 UTC) — Triggered manually via /fs-review by waynesun09. Re-posted the same truncation findings on both files. Verdict: CHANGES_REQUESTED.
Third review (21:04 UTC) — After the author pushed a fix adding a >= 300 fail-open guard. Acknowledged the truncation fix was resolved but still issued CHANGES_REQUESTED due to the persistent protected-path finding.

Human reviewer rh-hemartin approved the next morning and the PR merged.

Review quality assessment

Positive: The review agent's truncation finding was a genuine catch. The Compare API's 300-file cap is a real limitation with no pagination support, and the fail-open guard the author added was the correct remediation. This demonstrates strong review quality on infrastructure/CI changes.

Negative: Three CHANGES_REQUESTED verdicts for a 2-file human-authored CI fix created unnecessary friction. The protected-path finding (no linked issue) was a false positive in context — human-authored infrastructure fixes don't always originate from tracked issues. The second review duplicated findings already posted.

Existing issue coverage

All identified improvement areas are already tracked by open issues:

Duplicate findings across re-reviews: #1013, #1500
Protected-path severity on human PRs: #1551
Persistent CHANGES_REQUESTED after findings resolved: #1583, #1922
COMMENT verdict for human PRs with medium/low findings: #2115

No new proposals are warranted. Resolving the existing issues (particularly #1500, #1551, and #1583) would have reduced this PR's review cycle from 3 rounds to 1.

github-actions Bot deployed to site-preview June 24, 2026 16:36 View deployment

qodo-code-review Bot reviewed Jun 24, 2026

View reviewed changes

Comment thread .github/workflows/e2e.yml

fullsend-ai-review Bot suggested changes Jun 24, 2026

View reviewed changes

fullsend-ai-review Bot added component/ci CI pipelines and checks component/e2e End-to-end tests labels Jun 24, 2026

fullsend-ai-review Bot suggested changes Jun 24, 2026

View reviewed changes

github-actions Bot deployed to site-preview June 24, 2026 20:51 View deployment

fullsend-ai-review Bot suggested changes Jun 24, 2026

View reviewed changes

rh-hemartin approved these changes Jun 25, 2026

View reviewed changes

ralphbean added this pull request to the merge queue Jun 25, 2026

Merged via the queue into main with commit f502487 Jun 25, 2026
19 of 20 checks passed

ralphbean deleted the fix/merge-queue-path-filter branch June 25, 2026 13:58

qodo-code-review Bot mentioned this pull request Aug 18, 2026

chore(skills): add user forum What's New recap skill #6314

Merged

7 tasks

Uh oh!

Conversation

ralphbean commented Jun 24, 2026

Summary

Test plan

Uh oh!

qodo-code-review Bot commented Jun 24, 2026

PR Summary by Qodo

Uh oh!

github-actions Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Site preview

Uh oh!

fullsend-ai-review Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 24, 2026

Codecov Report

Uh oh!

qodo-code-review Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review

Findings

High

Review

Findings

High

Medium

Review

Findings

High

Medium

Review

Findings

High

Medium

Uh oh!

fullsend-ai-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

ralphbean Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

ralphbean Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

waynesun09 commented Jun 24, 2026

Uh oh!

fullsend-ai-review Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions Bot commented Jun 24, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 24, 2026 •

edited

Loading

qodo-code-review Bot commented Jun 24, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 24, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 24, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 24, 2026 •

edited

Loading

fullsend-ai-retro Bot commented Jun 25, 2026 •

edited

Loading