Skip to content

feat(ci): implement Vitest shards with merge-reports for e2e workflow #835

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steebchen merged 20 commits into from
Sep 15, 2025
Merged

feat(ci): implement Vitest shards with merge-reports for e2e workflow #835

Changes from 10 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
dcd5fe2
feat(ci): implement Vitest shards for e2e workflow
steebchen Sep 15, 2025
a40bd29
fix(ci): update e2e workflow to use run.yml and fix versions
steebchen Sep 15, 2025
07c526f
Revert "fix(ci): update e2e workflow to use run.yml and fix versions"
steebchen Sep 15, 2025
7404d2b
fix(ci): update artifact actions to v4
steebchen Sep 15, 2025
5e90d27
Refactor e2e.yml to use export for DATABASE_URL
steebchen Sep 15, 2025
13a23c2
Update vitest command to use pnpm in e2e workflow
steebchen Sep 15, 2025
5fed36e
fix(ci): handle test failures in e2e shards gracefully
steebchen Sep 15, 2025
d954672
fix(ci): fix merge-reports directory structure
steebchen Sep 15, 2025
367f72a
Remove cross-env-shell from E2E test command
steebchen Sep 15, 2025
a24ea0f
fix(ci): make upload-artifact step run unconditionally
steebchen Sep 15, 2025
11be9f4
Merge branch 'main' into terragon/e2e-vitest-shards-github-matrix
steebchen Sep 15, 2025
b828f9e
Disable fail-fast in e2e-shards workflow
steebchen Sep 15, 2025
5fd9a38
Merge branch 'main' into terragon/e2e-vitest-shards-github-matrix
steebchen Sep 15, 2025
8240f8e
Merge branch 'main' into terragon/e2e-vitest-shards-github-matrix
steebchen Sep 15, 2025
9b288b7
Update model name in API endpoint test
steebchen Sep 15, 2025
8ca0827
Update model name in API request
steebchen Sep 15, 2025
939af15
refactor(ci): consolidate merge-reports and final e2e jobs
steebchen Sep 15, 2025
e79abfa
refactor(ci): simplify merge-reports command
steebchen Sep 15, 2025
baf01bd
Merge branch 'main' into terragon/e2e-vitest-shards-github-matrix
steebchen Sep 15, 2025
8eeab3b
Merge branch 'main' into terragon/e2e-vitest-shards-github-matrix
steebchen Sep 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 117 additions & 5 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,121 @@ concurrency:
cancel-in-progress: true

jobs:
e2e:
uses: ./.github/workflows/run.yml
secrets: inherit
e2e-shards:
runs-on: ubuntu-latest
if: ${{ github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository }}

@coderabbitai coderabbitai Bot Sep 15, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Guard PR-only context in job condition to avoid expression errors on workflow_dispatch

Accessing github.event.pull_request.* when event_name != pull_request can throw. Gate the condition by event_name first.

-    if: ${{ github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository }}
+    if: ${{ github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository }}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if: ${{ github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository }}
if: ${{ github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository }}
🤖 Prompt for AI Agents 
In .github/workflows/e2e.yml around line 18, the job condition accesses
github.event.pull_request.* unguarded which can throw for non-pull_request
events (e.g. workflow_dispatch); update the if expression to first check the
event name (github.event_name == 'pull_request') and only then evaluate the
other pull_request-specific checks (actor and head.repo.full_name equality) so
the expression short-circuits and avoids errors on non-PR triggers.

with:
cmd: pnpm run setup && pnpm test:e2e
strategy:
matrix:
shard: [1, 2, 3, 4, 5]
steps:
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
- name: Checkout
uses: actions/checkout@v5
with:
persist-credentials: false

Comment thread
steebchen marked this conversation as resolved.
Resolved
- name: Collect versions
run: |
echo "nodejs_version=$(cat .tool-versions | grep 'nodejs' | cut -d ' ' -f 2)" >> "$GITHUB_ENV"
echo "pnpm_version=$(cat .tool-versions | grep 'pnpm' | cut -d ' ' -f 2)" >> "$GITHUB_ENV"

- uses: pnpm/action-setup@v4
with:
version: ${{ env.pnpm_version }}

- uses: actions/setup-node@v5
with:
node-version: ${{ env.nodejs_version }}
cache: pnpm

- name: Install dependencies
run: pnpm install --frozen-lockfile

- name: Setup and run e2e shard ${{ matrix.shard }}
run: |
set -eux
pnpm run setup
export DATABASE_URL=postgres://postgres:pw@localhost:5432/test
E2E_TEST=true pnpm vitest run -c vitest/vitest.e2e.config.mts --no-file-parallelism --shard=${{ matrix.shard }}/5 --reporter=blob --outputFile=e2e-results-${{ matrix.shard }}.blob || true
env:
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Comment thread
steebchen marked this conversation as resolved.
Resolved
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
INFERENCE_NET_API_KEY: ${{ secrets.INFERENCE_NET_API_KEY }}
TOGETHER_AI_API_KEY: ${{ secrets.TOGETHER_AI_API_KEY }}
GOOGLE_AI_STUDIO_API_KEY: ${{ secrets.GOOGLE_AI_STUDIO_API_KEY }}
CLOUD_RIFT_API_KEY: ${{ secrets.CLOUD_RIFT_API_KEY }}
MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
X_AI_API_KEY: ${{ secrets.X_AI_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
PERPLEXITY_API_KEY: ${{ secrets.PERPLEXITY_API_KEY }}
NOVITA_AI_API_KEY: ${{ secrets.NOVITA_AI_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
ALIBABA_API_KEY: ${{ secrets.ALIBABA_API_KEY }}
NEBIUS_API_KEY: ${{ secrets.NEBIUS_API_KEY }}
Z_AI_API_KEY: ${{ secrets.Z_AI_API_KEY }}
ROUTEWAY_API_KEY: ${{ secrets.ROUTEWAY_API_KEY }}

- name: Upload shard results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-${{ matrix.shard }}
path: e2e-results-${{ matrix.shard }}.blob
retention-days: 1
if-no-files-found: ignore

Comment thread
steebchen marked this conversation as resolved.
Resolved
merge-reports:
runs-on: ubuntu-latest
needs: e2e-shards
if: always()
steps:
- name: Checkout
uses: actions/checkout@v5
with:
persist-credentials: false

- name: Collect versions
run: |
echo "nodejs_version=$(cat .tool-versions | grep 'nodejs' | cut -d ' ' -f 2)" >> "$GITHUB_ENV"
echo "pnpm_version=$(cat .tool-versions | grep 'pnpm' | cut -d ' ' -f 2)" >> "$GITHUB_ENV"

- uses: pnpm/action-setup@v4
with:
version: ${{ env.pnpm_version }}

- uses: actions/setup-node@v5
with:
node-version: ${{ env.nodejs_version }}
cache: pnpm

- name: Install dependencies
run: pnpm install --frozen-lockfile

- name: Download all shard results
uses: actions/download-artifact@v4
with:
path: ./e2e-results
pattern: e2e-results-*

- name: Merge reports
run: |
# Create .vitest-reports directory and move blob files there
mkdir -p .vitest-reports
find ./e2e-results -name "*.blob" -exec mv {} .vitest-reports/ \;
# Merge reports
E2E_TEST=true pnpm vitest run -c vitest/vitest.e2e.config.mts --merge-reports

e2e:
runs-on: ubuntu-latest
needs: [e2e-shards, merge-reports]
if: always()
steps:
- name: Check shard results
run: |
if [[ "${{ needs.e2e-shards.result }}" == "success" && "${{ needs.merge-reports.result }}" == "success" ]]; then
echo "All e2e tests passed"
exit 0
else
echo "E2E tests failed"
exit 1
fi