Skip to content

ci(e2e): filter Vitest result comments #5593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jyaunches merged 1 commit into from
Jun 22, 2026
+67 −9
Merged

ci(e2e): filter Vitest result comments #5593

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 51 additions & 9 deletions .github/workflows/e2e-vitest-scenarios.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4487,21 +4487,57 @@ jobs:
prNumber = prs[0].number;
}

const requestedJobList = requestedJobs
.split(',')
.map((job) => job.trim())
.filter(Boolean);
const requestedJobSet = new Set(requestedJobList);
const selectiveDispatch =
requestedJobList.length > 0 || Boolean(requestedScenarios) || scenariosRejected || jobsRejected;
const emoji = { success: '✅', failure: '❌', cancelled: '⚠️', skipped: '⏭️' };
const entries = Object.entries(needs).sort(([a], [b]) => a.localeCompare(b));
const rows = entries.map(
const allEntries = Object.entries(needs).sort(([a], [b]) => a.localeCompare(b));
const missingRequested = selectorValidationPassed
? requestedJobList.filter((job) => !(job in needs))
: [];
const selectedEntries = requestedJobList.length > 0
? allEntries.filter(([name]) => requestedJobSet.has(name))
: selectiveDispatch
? allEntries.filter(
([name, { result }]) => result !== 'skipped' && name !== 'generate-matrix',
)
: allEntries;
const reportedEntries = selectedEntries.length > 0
? selectedEntries
: selectiveDispatch
? allEntries.filter(([, { result }]) => result !== 'skipped')
: allEntries;
Comment on lines +4509 to +4513

@coderabbitai coderabbitai Bot Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

reportedEntries fallback can incorrectly mark a no-signal run as passing.

On Line 4511, the selective-dispatch fallback repopulates from all non-skipped needs, which can pull generate-matrix back in as a passing entry after Line 4506 explicitly excluded it. That makes status resolve to a pass even when no selected jobs actually ran.

💡 Suggested fix 
-            const reportedEntries = selectedEntries.length > 0
-              ? selectedEntries
-              : selectiveDispatch
-                ? allEntries.filter(([, { result }]) => result !== 'skipped')
-                : allEntries;
+            const reportedEntries = selectedEntries.length > 0
+              ? selectedEntries
+              : selectiveDispatch
+                ? allEntries.filter(([name]) => name !== 'generate-matrix')
+                : allEntries;

Also applies to: 4521-4540

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e-vitest-scenarios.yaml around lines 4509 - 4513, The
fallback logic for reportedEntries in the selective dispatch case is incorrectly
including all non-skipped entries from allEntries, which can reintroduce the
generate-matrix entry that was explicitly excluded earlier. When
selectiveDispatch is true and selectedEntries is empty, the filter should
exclude generate-matrix along with skipped entries to prevent a no-signal run
from being marked as passing. Modify the selective dispatch fallback condition
to also filter out generate-matrix entries in addition to filtering out skipped
results, matching the explicit exclusion logic that appears on line 4506.

const rows = reportedEntries.map(
([name, { result }]) => `| ${name} | ${emoji[result] || '❓'} ${result} |`,
);
const ran = entries.filter(([, v]) => v.result !== 'skipped');
for (const name of missingRequested) {
rows.push(`| ${name} | ❓ not reported |`);
}

const ran = reportedEntries.filter(([, v]) => v.result !== 'skipped');
const passed = ran.filter(([, v]) => v.result === 'success');
const failed = ran.filter(([, v]) => v.result === 'failure');
const skipped = entries.filter(([, v]) => v.result === 'skipped');
const skipped = reportedEntries.filter(([, v]) => v.result === 'skipped');
const cancelled = ran.filter(([, v]) => v.result === 'cancelled');
const passingStatus = requestedJobList.length > 0
? '✅ All requested jobs passed'
: selectiveDispatch
? '✅ All selected jobs passed'
: '✅ All jobs passed';
const status =
failed.length > 0
failed.length > 0 || missingRequested.length > 0
? '❌ Some jobs failed'
: skipped.length > 0 && passed.length === 0
? '⚠️ No jobs ran'
: '✅ All jobs passed';
: cancelled.length > 0 && passed.length === 0
? '⚠️ Run cancelled — no signal'
: cancelled.length > 0 && passed.length > 0
? '⚠️ Some jobs cancelled — partial pass'
: skipped.length > 0 && passed.length === 0
? '⚠️ No selected jobs ran'
: passingStatus;

const lines = [
`### Vitest E2E Scenario Results — ${status}`,
Expand All @@ -4518,7 +4554,7 @@ jobs:
: requestedJobs
? `**Requested jobs:** \`${requestedJobs}\``
: '**Requested jobs:** _(default — all free-standing when no scenarios are requested)_',
`**Summary:** ${passed.length} passed, ${failed.length} failed, ${skipped.length} skipped`,
`**Summary:** ${passed.length} passed, ${failed.length} failed, ${cancelled.length} cancelled, ${skipped.length} skipped`,
'',
'| Job | Result |',
'|-----|--------|',
Expand All @@ -4528,6 +4564,12 @@ jobs:
const failedNames = failed.map(([name]) => name).join(', ');
lines.push('', `> **Failed jobs:** ${failedNames}. Check [run artifacts](${runUrl}) for logs.`);
}
if (missingRequested.length > 0) {
lines.push(
'',
`> **Missing requested jobs:** ${missingRequested.join(', ')}. The reporting workflow needs to include these jobs.`,
);
}

await github.rest.issues.createComment({
owner: context.repo.owner,
Expand Down
3 changes: 3 additions & 0 deletions test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -953,6 +953,9 @@ jobs:
"report-to-pr step must pass jobs through JOBS env",
"step 'Post Vitest scenario results to PR' run script must check selector validation before echoing selectors",
"step 'Post Vitest scenario results to PR' run script must omit rejected job selectors",
"step 'Post Vitest scenario results to PR' run script must filter reported entries for selective dispatches",
"step 'Post Vitest scenario results to PR' run script must report missing requested jobs",
"step 'Post Vitest scenario results to PR' run script must count cancelled jobs",
]),
);
} finally {
Expand Down
13 changes: 13 additions & 0 deletions tools/e2e-scenarios/workflow-boundary.mts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4398,6 +4398,19 @@ export function validateE2eVitestScenariosWorkflowBoundary(
"step 'Post Vitest scenario results to PR' run script must omit rejected scenario selectors",
);
}
if (!reportScript.includes("reportedEntries")) {
errors.push(
"step 'Post Vitest scenario results to PR' run script must filter reported entries for selective dispatches",
);
}
if (!reportScript.includes("missingRequested")) {
errors.push(
"step 'Post Vitest scenario results to PR' run script must report missing requested jobs",
);
}
if (!reportScript.includes("cancelled")) {
errors.push("step 'Post Vitest scenario results to PR' run script must count cancelled jobs");
}
if (!reportScript.includes("**Requested jobs:**")) {
errors.push(
"step 'Post Vitest scenario results to PR' run script must include **Requested jobs:**",
Expand Down