ci: enable e2e retries on merge queue runs - #41496
Conversation
|
Looks like this PR is ready to merge! 🎉 |
|
KevLehman
added
the
stat: QA assured
dionisio-bot
Bot
added
the
stat: ready to merge
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #41496 +/- ##
===========================================
+ Coverage 68.59% 68.61% +0.01%
===========================================
Files 4134 4134
Lines 160802 160802
Branches 29248 29281 +33
===========================================
+ Hits 110310 110341 +31
+ Misses 45385 45349 -36
- Partials 5107 5112 +5
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
WalkthroughThe CI workflow now enables up to two retries for UI CE, EE, and FIPS end-to-end test jobs when triggered by ChangesUI test retry configuration
Estimated code review effort: 1 (Trivial) | ~2 minutes Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Line 824: Update the test-ui-fips job condition near the retries configuration
to handle merge_group explicitly: allow the job based on the merge_group event
without evaluating github.event.pull_request.labels, while retaining the
existing fips-label check for pull request events and other current gating
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 05089729-fce1-4d37-b815-fe1bac7eb5a4
📒 Files selected for processing (1)
.github/workflows/ci.yml
📜 Review details
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2026-04-27T18:32:21.871Z
Learnt from: d-gubert
Repo: RocketChat/Rocket.Chat PR: 40321
File: .github/workflows/ci.yml:137-145
Timestamp: 2026-04-27T18:32:21.871Z
Learning: In .github/workflows/ci.yml, the `diff` step under `release-versions` intentionally uses a bash `if` with `gh pr diff ... | grep -q ...; then ... fi`. For non-`pull_request` workflow triggers where `GH_PR_NUM` can be empty, the `gh` command may fail, but the surrounding bash `if` is relied on to treat that failure as the condition being false and skip the `then` block, allowing the step/job to exit cleanly. Do not add extra guards for non-PR event types unless this failure/skip behavior is intentionally changed.
Applied to files:
.github/workflows/ci.yml
🔇 Additional comments (1)
.github/workflows/ci.yml (1)
662-662: LGTM!Also applies to: 753-753
| lowercase-repo: ${{ needs.release-versions.outputs.lowercase-repo }} | ||
| gh-docker-tag: ${{ needs.release-versions.outputs.gh-docker-tag }} | ||
| retries: ${{ (github.event_name == 'release' || github.ref == 'refs/heads/develop' || github.ref == 'refs/heads/master') && 2 || 0 }} | ||
| retries: ${{ (github.event_name == 'release' || github.event_name == 'merge_group' || github.ref == 'refs/heads/develop' || github.ref == 'refs/heads/master') && 2 || 0 }} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the job and the relevant condition/retry lines in the workflow.
rg -n -C 4 "test-ui-fips|contains\\(github\\.event\\.pull_request\\.labels|retries:" .github/workflows/ci.ymlRepository: RocketChat/Rocket.Chat
Length of output: 1973
🌐 Web query:
Rocket.Chat .github/workflows/ci.yml test-ui-fips merge_group contains(github.event.pull_request.labels)
💡 Result:
Based on the provided search results from the Rocket.Chat repository and general GitHub Actions practices, there is no evidence of a test-ui-fips job or configuration in the Rocket.Chat ci.yml file [1][2][3][4]. The repository's CI workflow focuses on standard UI and API tests (both Community and Enterprise editions) [1][2][3][4]. Regarding the technical concern about merge_group and contains(github.event.pull_request.labels), GitHub Actions merge_group events do not contain the pull_request object [5][6]. Consequently, using contains(github.event.pull_request.labels) within a job triggered by a merge_group event will fail because github.event.pull_request is undefined in that context [5][6]. To handle label-based logic in workflows that support both pull_request and merge_group events, best practices involve checking the event name [5][6]. For example: - if: github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'your-label') If a job must run for a merge group, it is common to either skip the label check or assume the necessary validation was completed during the original pull request check [5][6].
Citations:
- 1: https://github.com/RocketChat/Rocket.Chat/blob/e75965c0/.github/workflows/ci.yml
- 2: https://github.com/RocketChat/Rocket.Chat/blob/d33009ab/.github/workflows/ci.yml
- 3: https://github.com/RocketChat/Rocket.Chat/blob/3b003e6b/.github/workflows/ci.yml
- 4: https://github.com/RocketChat/Rocket.Chat/blob/ef22a22de58d4ec877569e5c17041e10742f6ae8/.github/workflows/ci.yml
- 5: https://github.com/clockworklabs/SpacetimeDB/blob/f04580d71f1e5143cecbc8a076c2cade6378f750/.github/workflows/check-merge-labels.yml
- 6: https://github.com/CMS-Enterprise/mint-app/blob/e99765d46a84d7065812589f91666f2bab07537f/.github/workflows/pr_label_checks.yml
🌐 Web query:
"contains(github.event.pull_request.labels.*.name, 'fips')" "Rocket.Chat" ".github/workflows/ci.yml"
💡 Result:
The expression contains(github.event.pull_request.labels.*.name, 'fips') is a GitHub Actions expression used to check if a pull request has a label named "fips" [1]. This pattern is commonly used in conditional logic within GitHub Actions workflow files to trigger specific jobs or steps [1]. While the "fips" label is used in Rocket.Chat's development context—such as in pull request #18661 regarding FIPS-compliant algorithms [2] and referenced in internal Jira tasks related to FIPS compliance [3]—there is no evidence that this specific contains expression is a hardcoded or standard part of the main Rocket.Chat .github/workflows/ci.yml file [4][5]. Rocket.Chat maintains FIPS-compliant Docker images for federal deployments [6], but the CI pipeline logic primarily relies on standard event triggers (e.g., pull_request, push, release) and input-driven workflows rather than label-based filtering for FIPS specifically [4][5][7]. If you are implementing this in a workflow, you must ensure that your job trigger includes the labeled or unlabeled types if you intend to run the workflow specifically when a label is added or removed [1]. Without these, the github.event.pull_request object may not contain the updated labels in all contexts [1].
Citations:
- 1: https://stackoverflow.com/questions/78078148/github-context-not-getting-labels-because-its-not-a-pull-request-how-can-i-d
- 2: [FIX] #18609 Use FIPS Algorithms for Connection IDs #18661
- 3: ci: add actionlint workflow #40010
- 4: https://github.com/RocketChat/Rocket.Chat/blob/e75965c0/.github/workflows/ci.yml
- 5: https://github.com/RocketChat/Rocket.Chat/blob/3b003e6b/.github/workflows/ci.yml
- 6: https://www.rocket.chat/blog/rocket-chat-8-6-virtru-integration-for-abac-and-fips
- 7: https://gitlab.ow2.org/RocketChat/Rocket.Chat/-/raw/ea8f99c082a17fe228de85813ae71ca10ed04b2f/.github/workflows/ci.yml
🌐 Web query:
site:github.com/RocketChat/Rocket.Chat/blob e75965c0 .github/workflows/ci.yml "test-ui-fips" "merge_group" "github.event.pull_request.labels"
💡 Result:
The query components refer to files and configuration logic within the Rocket.Chat repository (specifically at commit e75965c) [1][2][3]. 1. e75965c: This is a specific Git commit hash from the RocketChat/Rocket.Chat repository [1][2][3]. 2..github/workflows/ci.yml: This is the primary continuous integration workflow file for the project [1]. It defines the triggers for the CI pipeline, including pull_request, merge_group, and push events [1][4]. 3. test-ui-fips: This appears to be a specific CI job name or a referenced test suite configuration within the project's ecosystem, often utilized for FIPS-compliant UI testing [4][5]. 4. merge_group: This is a GitHub Actions trigger used to run workflows when a pull request is added to a merge group (part of GitHub's Merge Queue feature) [1][4]. 5. github.event.pull_request.labels: This is a GitHub Actions context expression used to access the labels associated with a pull request [4]. It is commonly used in conditional logic (if statements) within workflow steps or jobs to determine whether they should execute based on specific labels attached to a PR (e.g., triggering specialized tests only when a certain label is present) [4][5]. In the context of the Rocket.Chat repository, these elements are part of a complex CI/CD setup that utilizes reusable workflows (like ci-test-e2e.yml) [2][4] and conditional execution based on workflow events, branches, and PR metadata to manage testing, building, and deployment [1][4].
Citations:
- 1: https://github.com/RocketChat/Rocket.Chat/blob/e75965c0/.github/workflows/ci.yml
- 2: https://github.com/RocketChat/Rocket.Chat/blob/e75965c0/.github/workflows/ci-test-e2e.yml
- 3: https://github.com/RocketChat/Rocket.Chat/blob/e75965c0/.github/workflows/ci-code-check.yml
- 4: https://github.com/RocketChat/Rocket.Chat/blob/ef22a22de58d4ec877569e5c17041e10742f6ae8/.github/workflows/ci.yml
- 5: https://github.com/RocketChat/Rocket.Chat/blob/500d9a537bf81fe8cf950e989c4e737e16212565/.github/workflows/ci.yml
🌐 Web query:
GitHub Actions merge_group event payload pull_request object official docs
💡 Result:
The merge_group event in GitHub Actions does not contain a pull_request object [1]. This is a frequent point of confusion, as the merge_group event is distinct from the pull_request event [2][3][4]. While the pull_request event payload provides comprehensive details about a specific pull request, the merge_group event payload is focused on the temporary merge group branch created by the merge queue to validate changes [5][4][6]. Key points regarding the merge_group event payload: 1. Payload Structure: The top-level object for a merge_group event includes a merge_group field (an object) [6], but it does not include a pull_request field [1]. 2. Event Purpose: The merge_group event is triggered when a pull request is added to a merge queue [2][4]. It notifies workflows that they need to run status checks on the merge group branch [5][6]. 3. Data Availability: Because a single merge group can contain multiple queued pull requests [4][6], the event payload is structurally different from a standard pull_request event, which typically relates to a single, specific pull request [1]. 4. Accessing Information: If your workflow needs to interact with the pull request that triggered the queue, you cannot rely on the event payload to provide a pull_request object. Developers often have to use the GitHub API to fetch details about the pull requests included in the merge group if that information is required [1]. For further details on the specific fields available in the merge_group object (such as head_sha and base_sha), refer to the official GitHub Webhook events and payloads documentation [6].
Citations:
- 1: [BUG] merge_group tigger fails due to different event structure actions/dependency-review-action#843
- 2: https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows
- 3: https://docs.github.com/actions/using-workflows/events-that-trigger-workflows
- 4: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue
- 5: https://github.blog/changelog/2022-08-18-merge-group-webhook-event-and-github-actions-workflow-trigger/
- 6: https://docs.github.com/en/webhooks/webhook-events-and-payloads?actionType=auto_merge_enabled
Gate test-ui-fips for merge_group explicitly
.github/workflows/ci.yml still checks contains(github.event.pull_request.labels.*.name, 'fips'), which won’t work on merge_group runs because that event has no pull_request label context. If this job should run in the merge queue, split the condition by event name and skip the label lookup for merge_group.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml at line 824, Update the test-ui-fips job condition
near the retries configuration to handle merge_group explicitly: allow the job
based on the merge_group event without evaluating
github.event.pull_request.labels, while retaining the existing fips-label check
for pull request events and other current gating behavior.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
|
/jira ARCH-2228 |
3 participants
Summary
E2E test retries (
PLAYWRIGHT_RETRIES=2) were only enabled forreleaseevents and pushes todevelop/master. Merge queue runs come in asmerge_groupevents, so they ran with 0 retries and flaky tests could evict PRs from the queue.This adds
merge_groupto the retries condition for the three e2e jobs (test-ui,test-api-ee,test-ui-ee), so merge queue runs get the same 2 retries as develop.Summary by CodeRabbit
Task: ARCH-2285