Flaky tests: Report as a PR comment only, stop creating issues

.github/workflows/end2end-test.yml

@@ -144,9 +144,9 @@ jobs:
         runs-on: 'ubuntu-24.04'
         permissions:
             contents: read
+            # Needed to comment the flaky tests report on the pull request.
+            pull-requests: write
         timeout-minutes: 5
-        outputs:
-            has-flaky-test-report: ${{ !!steps.merge-flaky-tests-reports.outputs.artifact-id }}
         steps:
             - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
               with:
@@ -192,41 +192,20 @@ jobs:
                   path: playwright-report
                   if-no-files-found: ignore
 
-    report-to-issues:
-        name: Report to GitHub
-        needs: [merge-artifacts]
-        if: ${{ needs.merge-artifacts.outputs.has-flaky-test-report == 'true' }}
-        runs-on: 'ubuntu-24.04'
-        permissions:
-            contents: read
-            issues: write
-            pull-requests: write
-        timeout-minutes: 10
-        steps:
-            # Checkout defaults to using the branch which triggered the event, which
-            # isn't necessarily `trunk` (e.g. in the case of a merge).
-            - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-              with:
-                  ref: trunk
-                  show-progress: ${{ runner.debug == '1' && 'true' || 'false' }}
-                  persist-credentials: false
-
-            - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-              # Don't fail the job if there isn't any flaky tests report.
-              continue-on-error: true
+            # Flaky tests are only reported as a comment on the pull request. The
+            # merge step above only sets `artifact-id` when a report was produced,
+            # so both steps are skipped on runs without any flaky test.
+            - name: Download merged flaky tests report
+              id: download-flaky-tests-report
+              if: ${{ github.event_name == 'pull_request' && steps.merge-flaky-tests-reports.outputs.artifact-id }}
+              uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
               with:
                   name: flaky-tests-report
                   path: flaky-tests
 
-            - name: Setup Node.js and install dependencies
-              uses: ./.github/setup-node
-
-            - name: Npm build
-              run: npm run build -- --skip-types
-
             - name: Report flaky tests
+              if: ${{ steps.download-flaky-tests-report.outcome == 'success' }}
               uses: ./packages/report-flaky-tests
               with:
                   repo-token: '${{ secrets.GITHUB_TOKEN }}'
-                  label: '[Type] Flaky Test'
                   artifact-path: flaky-tests

.github/workflows/stale-issue-gardening.yml

@@ -33,13 +33,6 @@ jobs:
                       only-labels: 'Needs Testing'
                       remove-stale-when-updated: true
                       stale-issue-label: '[Status] Stale'
-                    - name: 'Flaky test issues without activity'
-                      message: 'This issue has gone 15 days without any activity.'
-                      days-before-stale: 15
-                      days-before-close: 1
-                      only-labels: '[Type] Flaky Test'
-                      remove-stale-when-updated: true
-                      stale-issue-label: '[Status] Stale'
 
         steps:
             - name: Update issues

docs/contributors/code/testing-overview.md

@@ -532,7 +532,7 @@ Every core block is required to have at least one set of fixture files for its m
 
 ### Flaky tests
 
-A test is considered to be **flaky** when it can pass and fail across multiple retry attempts without any code changes. We auto retry failed tests at most **twice** on CI to detect and report them to GitHub issues automatically under the [`[Type] Flaky Test`](https://github.com/WordPress/gutenberg/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22%5BType%5D+Flaky+Test%22) label via [`report-flaky-tests`](https://github.com/WordPress/gutenberg/tree/trunk/packages/report-flaky-tests) GitHub action. Note that a test that failed three times in a row is not counted as a flaky test and will not be reported to an issue.
+A test is considered to be **flaky** when it can pass and fail across multiple retry attempts without any code changes. We auto retry failed tests at most **twice** on CI to detect them, and report them, together with their errors, as a single comment on the pull request via the [`report-flaky-tests`](https://github.com/WordPress/gutenberg/tree/trunk/packages/report-flaky-tests) GitHub action. Note that a test that failed three times in a row is not counted as a flaky test and will not be reported, and that flaky tests are only reported on pull requests.
 
 ## PHP testing
 

packages/report-flaky-tests/CHANGELOG.md

@@ -2,4 +2,9 @@
 
 ## Unreleased
 
+### Breaking Changes
+
+-   Flaky tests are no longer reported to GitHub issues, and are no longer reported on pushes. The action now only comments the detected tests, together with their errors, on the pull request. The `label` input has been removed as a result.
+-   The action now runs straight from its TypeScript sources on Node's type stripping, instead of from a bundle. Relative imports must carry their `.ts` extension, and the syntax must be erasable — no enums, namespaces, parameter properties or decorators.
+
 -- Initial version of the package.

packages/report-flaky-tests/README.md

@@ -1,6 +1,8 @@
 # Report Flaky Tests
 
-A GitHub action to report flaky E2E tests to GitHub issues.
+A GitHub action to report flaky E2E tests as a comment on the pull request.
+
+It runs straight from the TypeScript sources in `src/`: the `node24` runtime strips the types on the fly, so there is no build step. Two constraints come with that — relative imports must carry their `.ts` extension, and the syntax must be erasable, which rules out enums, namespaces, parameter properties and decorators.
 
 **This package is still experimental and breaking changes could be introduced in future minor versions (`v0.x`). Use it at your own risks.**
 

packages/report-flaky-tests/action.yml

@@ -1,17 +1,17 @@
 name: 'Report flaky tests'
-description: 'Report flaky tests to GitHub issues'
+description: 'Report flaky tests as a comment on the pull request'
 inputs:
     repo-token:
         description: 'GitHub token'
         required: true
-    label:
-        description: 'The flaky-test label name'
-        required: true
-        default: 'flaky-test'
     artifact-path:
         description: 'The path of the downloaded artifact'
         required: true
         default: 'flaky-tests'
 runs:
+    # Node 24 strips the types on the fly, so the action runs straight from its
+    # TypeScript sources without a build step. This requires every relative
+    # import to carry its `.ts` extension, and the syntax to be erasable, which
+    # rules out enums, namespaces, parameter properties and decorators.
     using: 'node24'
-    main: 'build/index.cjs'
+    main: 'src/index.ts'

packages/report-flaky-tests/package.json

@@ -2,7 +2,7 @@
 	"name": "@wordpress/report-flaky-tests",
 	"version": "0.0.0",
 	"private": true,
-	"description": "A GitHub action to report flaky E2E tests to GitHub issues.",
+	"description": "A GitHub action to report flaky E2E tests as a comment on the pull request.",
 	"author": "The WordPress Contributors",
 	"license": "GPL-2.0-or-later",
 	"keywords": [
@@ -23,11 +23,12 @@
 		"node": ">=18.12.0",
 		"npm": ">=8.19.2"
 	},
-	"main": "build/index.mjs",
+	"type": "module",
+	"main": "src/index.ts",
 	"exports": {
 		".": {
 			"types": "./build-types/index.d.ts",
-			"require": "./build/index.mjs"
+			"import": "./src/index.ts"
 		},
 		"./package.json": "./package.json"
 	},
@@ -36,7 +37,6 @@
 		"@actions/core": "^1.9.1",
 		"@actions/github": "^5.0.1",
 		"@jest/test-result": "^30.4.1",
-		"@octokit/types": "^6.34.0",
 		"@octokit/webhooks-types": "^5.8.0",
 		"@playwright/test": "^1.62.1",
 		"jest-message-util": "^30.4.1"

packages/report-flaky-tests/src/__tests__/__snapshots__/markdown.test.ts.snap

@@ -68,36 +68,3 @@ exports[`formatTestErrorMessage should format test error message for jest-circus
           at runMicrotasks (<anonymous>)
 "
 `;
-
-exports[`parseIssueBody should parse issue body 1`] = `
-"<!-- __META_DATA__:{"failedTimes":5,"totalCommits":95} -->
-**Flaky test detected. This is an auto-generated issue by GitHub Actions. Please do NOT edit this manually.**
-
-## Test title
-Test title
-
-## Test path
-\`test/e2e/specs/test-title.spec.js\`
-
-## Errors
-<!-- __TEST_RESULTS_LIST__ -->
-<!-- __TEST_RESULT__ --><time datetime="2020-08-01T00:00:00.000Z"><code>[2020-08-01T00:00:00.000Z]</code></time> Test passed after 1 failed attempt on <a href="https://github.com/WordPress/gutenberg/actions/runs/2282393879"><code>trunk</code></a>.<!-- /__TEST_RESULT__ -->
-<!-- __TEST_RESULT__ --><details>
-<summary>
-	<time datetime="2020-05-10T00:00:00.000Z"><code>[2020-05-10T00:00:00.000Z]</code></time> Test passed after 2 failed attempts on <a href="https://github.com/WordPress/gutenberg/actions/runs/2297863316"><code>try/some-branch</code></a>.
-</summary>
-
-\`\`\`
-Error:
-  Oops...
-  Oops again
-\`\`\`
-</details><!-- /__TEST_RESULT__ -->
-
-<!-- __TEST_RESULT__ -->Some broken log.
-Oops.
-<!-- /__TEST_RESULT__ -->
-
-<!-- /__TEST_RESULTS_LIST__ -->
-"
-`;