onTestFailure not being called when test is retried (6.9.6)

[jineshqa]

My Test :
@listeners({ ScreenshotListener.class })
public class TestRetryLogic {
@test(retryAnalyzer = Retry.class)
public void testRetryLogic() {
fail("Failing");
}
}

My Retry Analyzer :
public class Retry implements IRetryAnalyzer {
private final int maxRetryCountPerTest = 1;
private int retryCountPerTest = 0;

public boolean retry(ITestResult result) {
    if(retryCountPerTest < maxRetryCountPerTest) {
        retryCountPerTest++;
        return true;
    }
    return false;
}

}

My Listener :
public class ScreenshotListener extends TestListenerAdapter {

@Override
public void onTestFailure(ITestResult tr) {
    System.out.println("Test FAILED");
}

@Override
public void onTestSkipped(ITestResult tr) {
    System.out.println("Test SKIPPED");
}

@Override
public void onTestStart(ITestResult result) {
    System.out.println("Test STARTED");
}

}

My output:
Test STARTED
Test SKIPPED
Test STARTED
SKIPPED: testRetryLogic
java.lang.AssertionError: Failing

I would have expected :

Test STARTED
Test SKIPPED
Test STARTED
Test FAILED (Missing from above)
SKIPPED: testRetryLogic
java.lang.AssertionError: Failing

Is this a bug? Or is this expected?

Thank you.

[juherr]

If the behavior was different before, it is a bug.

[jineshqa]

It certainly was. I have code to take screenshot and add that using reporter.log in testng results. Now, none of that is working :-(

[juherr]

Ok, thank. I will have a look asap.

[juherr]

@jineshkenandy I tried to reproduce your issue without success. See #860 and tell me if you see something bad.

[jineshqa]

@juherr Could you put a print statement inside your onTestFailure method? Count has never been the issue it counts correctly for some reason.

Another thing I observed is that you are calling testng from code and not xml. All of us who faced the issue are calling it from XML.

Thanks,
Jinesh

[juherr]

No need to print: the method is called and stored.
But you can try on your own side by pulling my branch.

+1 about xml, I will try. It should not change anything but I've already seen xml only problems.

[jineshqa]

@juherr
I set my retry analyzer using below code. I noticed that when I set retry analyzer print statement inside onTestFailure is not executed. However if retry analyzer is not set print statement is executed correctly.

@BeforeTest(alwaysRun = true)
public void beforeTest(ITestContext context) {
    for (ITestNGMethod method : context.getAllTestMethods()) {
        method.setRetryAnalyzer(new utils.common.RetryAnalyzer());
    }
}

[juherr]

It is really an uncommon way the initialize it? Can I ask why do you do it?
IMO, you should use annotation transformer instead.

BTW, about the print, can we suppose it is a display issue?

However if retry analyzer is not set print statement is executed correctly

I don't understand what you want to say.

[juherr]

@jineshkenandy I updated the sample with what I understood of your comments, but I confirm the result is still the same for me:

START
SKIPPED
START
FAILURE

[szaluk]

I am initializing my Retry Analyzer the same way as @jineshkenandy I am seeing the same thing. In a previous version of TestNG I handled the retries in the onTestFailed() method but when I updated to version 6.9.6 I noticed that wasn't getting called anymore. I have since had to add code to the onTestSkipped() and the onTestFailedButWithinSuccessPercentage() to handle my retries. It's working fine now but I was surprised by the change.

[juherr]

@jineshkenandy @szaluk Could you try the latest testng version (6.9.9) and tell me if you still observe the same problem?

Because, as proved by #860, the problem doesn't occur on the master branch.

[szaluk]

It does seem to be fixed in 6.9.9. I created a very simple project to test it out that does 2 retries.

This is what I am seeing in 6.9.6:
Inside onTestSkipped()
Inside onTestSkipped()
Inside onTestFailedButWithinSuccessPercentage()

This is what I am seeing in 6.9.9:
Inside onTestSkipped()
Inside onTestSkipped()
Inside onTestFailure()

I will upgrade to 6.9.9. Thanks!

--Steve

BTW, I looked at the change log for TestNG and there is nothing in there for the releases > 6.9.6 that mentions a fix for retried tests: https://github.com/cbeust/testng/blob/master/CHANGES.txt

Am I looking in the wrong spot?

[jineshqa]

@juherr I meant that if I do not use the retry analyzer at all and let test just fail. The print statement inside onTestFailure gets executed.

@szaluk Thanks for pitching in. I am having exact same issue. I'll upgrade and get back with my findings.

[jineshqa]

@juherr Same result with 6.9.9 as well and I moved to annotation transformer.

[jineshqa]

@szaluk Could you confirm that 6.9.9 is working fine for you on actual project and not test project?

[szaluk]

Sure. I will be able to try that tomorrow. I will let you know what I see.

Thanks,
Steve

[szaluk]

@jineshkenandy - It does indeed seem to be working for me with 6.9.9 in my main project. This is how I am setting up my retry anaylzer:

@BeforeClass(alwaysRun = true)
public void baseTestClassSetup(ITestContext testContext) {
    for(ITestNGMethod testNGMethod : testContext.getAllTestMethods()) {
        if(testNGMethod.getRetryAnalyzer() == null)
            testNGMethod.setRetryAnalyzer(new RetryAnalyzer());
    }
}

This is how you were originally doing it before you changed over to annotation transformer.

[jineshqa]

@szaluk Are you using testng to run selenium webdriver?

[szaluk]

@jineshkenandy - Yes. It's a Maven based project using Java 7 and Selenium WebDriver 2.48.2

[jineshqa]

@szaluk I am using Selenium WebDriver 2.48.2 with testNg, no luck whatsoever.

[juherr]

Guys, should we understand the issue is fixed with 6.9.9?

[jineshqa]

@juherr Please mark it as closed. Thanks for all the help.

[ghost]

Hi,

I can reproduce it in 6.9.9 as well.

Best regards,

Baubak

From: Steven Zaluk <[email protected]mailto:[email protected]>
Reply-To: cbeust/testng <[email protected]mailto:[email protected]>
Date: Friday 6 November 2015 at 18:51
To: cbeust/testng <[email protected]mailto:[email protected]>
Subject: Re: [testng] onTestFailure not being called when test is retried (6.9.6) (#857)

It does seem to be fixed in 6.9.9. I created a very simple project to test it out that does 2 retries.

This is what I am seeing in 6.9.6:
Inside onTestSkipped()
Inside onTestSkipped()
Inside onTestFailedButWithinSuccessPercentage()

This is what I am seeing in 6.9.9:
Inside onTestSkipped()
Inside onTestSkipped()
Inside onTestFailure()

I will upgrade to 6.9.9. Thanks!

--Steve

BTW, I looked at the change log for TestNG and there is nothing in there for the releases > 6.9.6 that mentions a fix for retried tests: https://github.com/cbeust/testng/blob/master/CHANGES.txt

Am I looking in the wrong spot?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/857#issuecomment-154484237.