Flaky test marking and retrying

[lukasholzer]

Clear and concise description of the problem

Often some tests tend to be flaky which is a nightmare in large CI/CD pipelines as you have to retrigger and rerun.

There is not always the time to fix those flaky tests (what would be the ideal case - no doubt)
But in reality it happens.

It would be nice to have a retry mechanism that does not block CI runs.

Suggested solution

I propose a feature that would mark tests as flaky (and then it would retry them up to a max that is specified when it fails)

like:

test.flaky('This test is marked as flaky', () => { /* some flaky logic */ })

in the vitest.config.ts it could be then configured how often a flaky test should be retried on failure (default to 3 times)

/// <reference types="vitest" />
import { defineConfig } from 'vite';

export default defineConfig({
  test: {
    flakyTestRetries: 4, // default would be 3
  },
})

the test would then be rerun if it fails up to the maximum specified.

It would only do that for tests that are marked as test.flaky (Therefore it needs an incentive of the user to say this test is really flaky I know but I don't have time to fix it now)

In the summary of the end it could then be written 3 flaky tests detected in addition to how many are passed or skipped.

Alternative

Not implement this feature

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[kwangure]

There should also be a way to retry ALL failed tests without needing special syntax like test.flaky(....), similar to @playwright/test retries. This is especially convenient when you have several flaky tests for one reason or the other.

[bluwy]

On top of this, if beforeEach provides a hint that it's currently a retry (or the current retry index), it would help to setup "debug features" for more verbose test results, e.g. with Playwright I can turn on tracing when retrying.

[kwangure]

Another features of Playwright I've missed in Vitest is the ability to distinguish initial runs from retries. Like in the @playwright/test retries docs demo, it allows clearing cache on the server or browser for example.