Flaky tests: Report as a PR comment only, stop creating issues

[youknowriad]

What?

Stops reporting flaky E2E tests to GitHub issues. Flaky tests are now reported only as a comment on the pull request, posted from the job that already fans the shards in.

Four things go away:

1. Issue creation — no more [Type] Flaky Test issues.
2. The cleanup job — the Flaky test issues without activity matrix entry in stale-issue-gardening.yml, which marked those issues stale after 15 days and closed them a day later.
3. Push reporting — the action used to post a comment on the commit for pushes to trunk. That path is gone.
4. The Report to GitHub job, and the build step it existed for.

Why?

The flaky-test issues were mostly auto-generated churn. Few of them got acted on before the gardening job closed them again, so the two jobs largely existed to feed and then clean up after each other. The signal people actually use is the comment on the PR that surfaced the flake.

Removing the issue logic left the action small enough that its dedicated job stopped making sense. From a run where it fired:

Set up job                    1s
checkout                      4s
download-artifact             0s
Setup Node.js                24s
Npm build                    65s   ← full monorepo build, to compile one small action
Report flaky tests           13s
                            ---
                            110s

How?

Workflows

• end2end-test.yml: the report-to-issues job is gone. Its two useful steps moved into Merge Artifacts, which already fans the shards in, already checks out, and already installs dependencies — it only needed pull-requests: write. stale-issue-gardening.yml loses the flaky-test matrix entry; the two other gardening entries are untouched.

The artifact fan-in stays, and is load-bearing: matrix jobs can't aggregate through outputs (all legs write the same key and one arbitrary value wins), and nine shards racing to upsert one comment would each fail to find an existing one and create their own.

No build step

The action now runs straight from its TypeScript sources — action.yml points main at src/index.ts and the node24 runtime strips the types on the fly. Two constraints come with that, documented in action.yml and the README:

• relative imports must carry their .ts extension (./run → ./run.ts), which is also why allowImportingTsExtensions is now set;
• the syntax must be erasable — no enums, namespaces, parameter properties or decorators.

Action (packages/report-flaky-tests)

• run.ts no longer fetches, creates, or reopens issues; it maps the artifact straight to a PR comment, and bails out early on non-pull_request events.
• github-api.ts drops fetchAllIssuesLabeledFlaky, findMergeBaseCommit, createIssue, updateIssue, and createCommentOnCommit.
• markdown.ts drops the issue-body renderer and the parser that read state back out of it.
• The label input, the @octokit/types dependency, and the flaky-issues.json fixture are removed.

One behavior change worth a look: the comment now carries the error output. It previously listed issue numbers (- #1 in path``) with the stack traces in the issue body. With issues gone that detail had nowhere to go, so each test renders as a collapsed <details>:

🔍  Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/100
📝  Reported tests:
<details>
<summary>should copy only partial selection of text blocks in <code>/test/e2e/specs/…</code>, passed after 1 failed attempt.</summary>
… stack trace …
</details>

Side effect: a flaky test with no pre-existing issue used to be silently dropped from the comment, so comments only listed already-known flakes. Now every flake in the run is listed.

Testing Instructions

CI on this PR exercises it end to end — if a test goes flaky, Merge Artifacts should post a single comment listing the tests with their errors, and no issue should be created.

Locally:

npm run test:unit packages/report-flaky-tests

7 tests pass, covering the PR comment, the non-PR skip, and the empty-artifact case.

To confirm the action really runs without a build, from the repo root:

env 'INPUT_REPO-TOKEN=fake' 'INPUT_ARTIFACT-PATH=/tmp/flaky' GITHUB_EVENT_NAME=pull_request GITHUB_EVENT_PATH=/tmp/event.json GITHUB_REPOSITORY=WordPress/gutenberg GITHUB_RUN_ID=1 node packages/report-flaky-tests/src/index.ts

With a flaky-test JSON in /tmp/flaky and a PR payload in /tmp/event.json, this reaches the GitHub API and fails only on the fake token — no compilation involved. Note this needs Node 24; .nvmrc pins 20, which has no type stripping, so use nvm use 24 for that check. It has no bearing on CI, where the action runs on the node24 runtime, or on the unit tests, which Jest transpiles.

Testing Instructions for Keyboard

N/A — CI-only change, no user interface.

Screenshots or screencast

N/A

Use of AI Tools

This PR was authored with Claude Code (Opus 5) — the workflow changes, the action rewrite, the test updates, and this description. I reviewed the result and am responsible for it. Unit tests, tsc, lint and format were run locally and pass, and the no-build execution path was verified by actually running the entrypoint as described above.

Two things surfaced during review that I have not changed and would like a second opinion on:

• Merge Artifacts now holds pull-requests: write while running npm ci from the PR's checkout, where the old job was isolated on ref: trunk. For pull_request events from forks the GITHUB_TOKEN is read-only regardless, and same-repo branches already carry write access, so I believe this is inert — but it is a real change in posture. Keeping a separate job with no checkout at all would preserve the isolation and still avoid the build, at the cost of one extra job.
• formatTestErrorMessage still has a jest-circus branch, but the only producer (flaky-tests-reporter.ts) hardcodes runner: '@playwright/test'. It looks like dead code left over from the Puppeteer suite, and it is the only reason jest-message-util and @jest/test-result are dependencies. I left it in place rather than expand the scope of this PR.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @riad@Riads-MacBook-Air.local.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: riad@Riads-MacBook-Air.local.

Co-authored-by: youknowriad <youknowriad@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[github-actions]

Size Change: 0 B

Total Size: 7.81 MB

_{compressed-size-action}