Documentation: show how to rerun failed tests

[ericdriggs]

Purpose: documentation for rerunning failed tests

Use case: large suites of api tests with downstream service dependencies on other services have a high probability of failure. Option to rerun failed tests can make test management orders of magnitude easier and can be useful for distinguishing between logical errors and race conditions.

see:

• implement way to retry tests that have failed #998
• https://stackoverflow.com/questions/53912893/how-to-rerun-failed-features-in-karate

Code examples

@Test
    void testParallel() {
        Results results = Runner.path("classpath:demo")
                .outputCucumberJson(true)
                .karateEnv("demo")
                .parallel(5);
        results = rerunFailedTests(results,3);
        generateReport(results.getReportDir());
        assertTrue(results.getFailCount() == 0, results.getErrorMessages());
    }

 //note: requires log variable
 @SuppressWarnings("SameParameterValue")
    public static Results rerunFailedTests(Results results, int rerunCount) {

        for (int i = 1; i <= rerunCount; i++) {

            //if no failures, no need to retry
            List<ScenarioResult> failed = results.getScenarioResults().filter(ScenarioResult::isFailed).collect(Collectors.toList());
            if (failed.isEmpty()) {
                return results;
            }

            final Suite suite = results.getSuite();
            final int attempt = i;
            log.info("{} failed tests. retry attempt: {}", failed.size(), attempt);

            Set<ScenarioResult> retriedResults = new ConcurrentSkipListSet<>();

            //parallel retries
            failed.parallelStream().forEach(scenarioResult -> {
                Scenario scenario = scenarioResult.getScenario();
                log.info("retrying scenario: {}, attempt: {}", scenario.getName(), attempt);
                ScenarioResult sr = suite.retryScenario(scenario);
                retriedResults.add(sr);
            });

            //update results synchronously in case of race conditions
            for (ScenarioResult sr : retriedResults) {
                results = results.getSuite().updateResults(sr);
            }
        }
        //return after max rerunCount attempts
        return results;
    }

[sadiqkassamali]

peter has already mentioned it here
https://github.com/karatelabs/karate/blob/develop/karate-core/src/test/java/com/intuit/karate/core/retry/RetryTest.java#L35

i++ will make it continues loop without break; ideally you would want to run it once

[ericdriggs]

@sadiqkassamali
The example in the code shows a single retry for a single test.
The example I provided is a drop-in function which provides multiple retries with parallel execution.

[sadiqkassamali]

i am doing this this way.
List failed = results.getScenarioResults().filter(sr -> sr.isFailed())
.collect(Collectors.toList());

        try {
            boolean rerun = true;
            while(rerun)
                if (!failed.isEmpty()) {

                log.info("Fail size: + " + failed.size());
                log.info("May have found failures for Rerun..checking");
                for (int i = 0; i < failed.size(); i++) {
                    log.info("Total number of test case that failed, which will be rerun: " + failed.size());
                    Scenario scenario = failed.get(i).getScenario();
                    ScenarioResult sr = results.getSuite().retryScenario(scenario);
                    results = results.getSuite().updateResults(sr);
                    collector.checkThat(results.getFailCount(), equalTo(0));
                    rerun = false;
                    break;
                }

            } else {
                log.info("Fail size: + " + failed.size());
                log.info("Wow No Failures in the first run");
                collector.checkThat(results.getFailCount(), equalTo(0));
                rerun = false;
                break;
            }
        } catch (Exception e) {
            log.info("Error occurred Sending out email" + e.getMessage());

        }

[ericdriggs]

@sadiqkassamali

1. possible infinite loop
2. single threaded

[sadiqkassamali]

thanks for letting me know appreciate it, i stand corrected

[sturose]

@ericdriggs I would probably do:

    // parallel retries
    List<ScenarioResult> retriedResults =
        failed.parallelStream()
            .map(
                scenarioResult -> {
                  Scenario scenario = scenarioResult.getScenario();
                  log.info("retrying scenario: {}, attempt", scenario.getName());
                  return suite.retryScenario(scenario);
                })
            .toList();

[ericdriggs]

@sturose If you want a single retry

[sturose]

I'm not sure I follow. I'm just saying that instead of doing

            Set<ScenarioResult> retriedResults = new ConcurrentSkipListSet<>();

            //parallel retries
            failed.parallelStream().forEach(scenarioResult -> {
                Scenario scenario = scenarioResult.getScenario();
                log.info("retrying scenario: {}, attempt: {}", scenario.getName(), attempt);
                ScenarioResult sr = suite.retryScenario(scenario);
                retriedResults.add(sr);
            });

it reads a bit cleaner to do

    // parallel retries
    List<ScenarioResult> retriedResults =
        failed.parallelStream()
            .map(
                scenarioResult -> {
                  Scenario scenario = scenarioResult.getScenario();
                  log.info("retrying scenario: {}, attempt", scenario.getName());
                  return suite.retryScenario(scenario);
                })
            .toList();

They are functionality equivalent.

[ptrthomas]

tagging @bischoffdev since he maintains https://github.com/trivago/cluecumber - that supports generating reports for karate runs. I saw a recent update that improves support for when tests are re-tried - and perhaps Benjamin you have some experience with re-trying failed tests in karate ...