Tests always pass in combination with Arquillian

[alinuxexpert]

I'm not sure if this is a TestNG or an arquillian issue. However, downgrading TestNG solves the problem, so I'm attributing it to TestNG.

In TestNG version 6.9.13.3 to 6.11 arquillian always shows my tests as passed, even if I do assertTrue(false). Downgrading to TestNG version 6.9.10 solves the issue. I also found this, proving that I'm not the only one:

https://stackoverflow.com/questions/42558563/no-failure-with-it-tests-when-exception-is-thrown-arquillian-testng-maven

Here's a minimal example (full eclipse 4.7 project):
test.zip
Just run the only TestNG class. The test will pass, even though it shouldn't. Then change the TestNG version in the pom.xml to 6.9.10, run the test again, and it will fail.

Here goes the list of my project dependencies, so you know my configuration:

<dependency>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
      <version>2.5</version>
    </dependency>
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-lang3</artifactId>
      <version>3.6</version>
    </dependency>
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-text</artifactId>
      <version>1.1</version>
    </dependency>
    <dependency>
      <groupId>org.mariadb.jdbc</groupId>
      <artifactId>mariadb-java-client</artifactId>
      <version>2.1.0</version>
    </dependency>
    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>6.9.10</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.glassfish.main.extras</groupId>
      <artifactId>glassfish-embedded-all</artifactId>
      <version>4.1.2</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.jboss.arquillian.container</groupId>
      <artifactId>arquillian-glassfish-embedded-3.1</artifactId>
      <version>1.0.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.jboss.arquillian.testng</groupId>
      <artifactId>arquillian-testng-container</artifactId>
      <version>1.1.13.Final</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.jboss.arquillian</groupId>
      <artifactId>arquillian-bom</artifactId>
      <version>1.1.13.Final</version>
      <scope>import</scope>
      <type>pom</type>
    </dependency>

[krmahadevan]

@linex83 - I am not familiar with arquillian. I just downloaded the zip file, unzipped it and imported it into my IntelliJ IDE.

I found only test.NameProviderTest as a test class. But I am not able to figure out how do I run it, because it prompts me for some configurations.

Can you please help add some detailed instructions on how to get this going in an IDE ?

[juherr]

Could you just try with 6.12 from jcenter? It contains a fix asked by arquillian and it may be this one.

[alinuxexpert]

Thank you very much for the quick responses!

@juherr I tried version 6.12, it didn't solve the problem, though.

@krmahadevan You can run the tests from the shell with mvn test. With Eclipse you can just right click the class test.NameProviderTest and select "Run As > TestNG Test" (requires Eclipse plugin TestNG for Eclipse). Sorry I'm not familiar with IntelliJ IDE at all - but you can probably run mvn test from there as well.

[alinuxexpert]

Just to add further details: I tried running testng with arquillian and WildFly (instead of GlassFish) and got the same results.

[phillipross]

We've been stuck at v6.10 for awhile due to this problem. I didn't try v6.12 yet, but I'm guessing that the issue has not been resolved yet.

[krmahadevan]

@phillipross - Yes I guess this is still a problem. I had a go at this issue in the recent past, and still haven't figured out the overall flow that Arquillian has. I know TestNG, but since I dont know how Arquillian works, I keep getting lost when I attempt at tracing the flow. Will give this another go over the weekend and see what I can come up with. In the meantime, if you can help me point to any documentation that would help me quickly understand Arquillian and help me debug, it would be great!

[phillipross]

@krmahadevan - I'm not familiar with any Arquillian documentation outside of the guides published on their site. I can try to put together a very minimal test case that illustrates proper behavior when testng v6.9.10 is used and improper behavior when anything later than 6.9.10 is used. I still have to test v6.12 myself which I'm about to do right now.

[phillipross]

@krmahadevan I've verified that v6.12 still suffers from the same problem. I'm attaching a test case that hopefully helps troubleshoot the problem.
arquillian-testng-testing.zip

The zip file contains a self-contained maven project. If you run mvn clean test the project should build and try to run the test, but is should fail with an error message stating that it should fail the build. The test method uses Assert.fail() method to generate this error. And the failure is correct behavior.

You can control the TestNG version by setting the dependency.testng.version property. Try setting it to a newer version, and the build will NOT fail, even though Assert.fail() is used. Examples:

mvn -Ddependency.testng.version=6.9.10 clean test # This should properly fail
mvn -Ddependency.testng.version=6.10 clean test   # This won't fail, but it should!
mvn -Ddependency.testng.version=6.11 clean test   # This won't fail, but it should!
mvn -Ddependency.testng.version=6.13.1 clean test   # This won't fail, but it should!

I'm not an Arquillian expert, but I'm familiar with the basics and can attempt to answer any questions you might have. I appreciate the attention to this issue!

[phillipross]

@krmahadevan when you said you dealt with this issue in the past, was it a separate github issue that was already reported, seemingly fixed, and then closed? Or did you mean you tried tracing through in an effort to address this github issue? thanks!

[krmahadevan]

@phillipross - I was attempting at trying to reproduce this issue so that I may try to fix this. There's no previous issue that I am aware of which talks about the same problem.

I still haven't gotten around to this issue. Kind of stuck with stuff at the personal front and things are a bit hectic at the work front as well. So please bear with the delay.

[phillipross]

I'm happy to report that the as of testng v6.13.1 and arquillian-core v1.1.15.Final this issue seems to have been addressed. I'll update the test case code and attach it to this issue for posterity.

[phillipross]

arquillian-testng-testing.zip

mvn -Ddependency.testng.version=6.9.10 clean test # This should properly fail
mvn -Ddependency.testng.version=6.10 clean test   # This won't fail, but it should!
mvn -Ddependency.testng.version=6.11 clean test   # This won't fail, but it should!
mvn -Ddependency.testng.version=6.13.1 clean test   # This won't fail, but it should!
mvn -Ddependency.testng.version=6.13.1 -Ddependency.arquillian.version=1.1.13.Final clean test   # This won't fail, but it should!
mvn -Ddependency.testng.version=6.13.1 -Ddependency.arquillian.version=1.1.14.Final clean test   # This won't fail, but it should!
mvn -Ddependency.testng.version=6.13.1 -Ddependency.arquillian.version=1.1.15.Final clean test   # This DOES fail... the issue has been resolved!

[krmahadevan]

@phillipross Thank you so much for taking the time to revisit this issue and confirm that this is now working.
Closing this issue based on @phillipross comments.

[chyt]

I'm also experiencing a similar issue. For me, the tests run properly when using TestNG version 6.9.10 and Arquillian 1.1.13.Final. However, the tests fail to run with TestNG 6.13.1 and Arquillian 1.1.15.Final. I've also tried TestNG 6.13.1 and Arquillian 1.2.0.Final, but the tests also fail to run.

[juherr]

@chyt But what is the expected behavior? Failed or succeed tests?

[chyt]

@juherr Sorry for being ambiguous in my previous post, when I said "fail" I meant "fail to execute." Similar to OP, I'm experiencing that TestNG is telling me that the tests are succeeding, when in fact they did not run. I've updated my previous post to clarify this.

[juherr]

The issue may be related to arquillian/arquillian-core#134 but it should work with TestNG 6.13+ + Arquillian 1.1.15+.

Ping @bartoszmajsak Any idea before I try to digg?

[bartoszmajsak]

If @chyt can provide reproducer I can take a look.

@chyt if the tests fail to run, does it result with some stack trace which you could share?

I need more context to start digging.

[chyt]

I'm using Maven to run TestNG with Arquillian using the WLP managed container. There's no stacktrace, only standard Arquillian output:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running application.rest.TNGTransactionRestEndpointIT
Launching LibertyProjectServer (WebSphere Application Server 17.0.0.2/wlp-1.0.17.cl170220170523-1818) on Java HotSpot(TM) 64-Bit Server VM, version 1.8.0_131-b11 (en_US)
[AUDIT   ] CWWKE0001I: The server LibertyProjectServer has been launched.
[AUDIT   ] CWWKE0100I: This product is licensed for development, and limited production use. The full license terms can be viewed here: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/base_ilan/ilan/17.0.0.2/lafiles/en.html
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /Users/ctianus.ibm.com/Spring/liberty-runner-app/target/wlp/usr/servers/LibertyProjectServer/configDropins/defaults/install_apps_configuration_1491924271.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /Users/ctianus.ibm.com/Spring/liberty-runner-app/target/wlp/usr/servers/LibertyProjectServer/configDropins/defaults/net.wasdev.wlp.starters.persistence.server-snippet-0.0.3.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /Users/ctianus.ibm.com/Spring/liberty-runner-app/target/wlp/usr/servers/LibertyProjectServer/configDropins/defaults/net.wasdev.wlp.starters.rest.server-snippet-0.0.3.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /Users/ctianus.ibm.com/Spring/liberty-runner-app/target/wlp/usr/servers/LibertyProjectServer/configDropins/defaults/net.wasdev.wlp.starters.web.server-snippet-0.0.3.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/50d712b0-768a-4ecb-9944-e457a6191166/
[AUDIT   ] CWWKZ0001I: Application 50d712b0-768a-4ecb-9944-e457a6191166 started in 0.452 seconds.
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/myLibertyApp/
[AUDIT   ] CWWKZ0001I: Application test started in 2.215 seconds.
[AUDIT   ] CWWKF0012I: The server installed the following features: [ejbLite-3.2, servlet-3.1, jndi-1.0, json-1.0, localConnector-1.0, jpaContainer-2.1, jsonp-1.0, jdbc-4.1, jaxrs-2.0, jpa-2.1, jaxrsClient-2.0].
[AUDIT   ] CWWKF0011I: The server LibertyProjectServer is ready to run a smarter planet.
[AUDIT   ] CWWKT0017I: Web application removed (default_host): http://localhost:9080/50d712b0-768a-4ecb-9944-e457a6191166/
[AUDIT   ] CWWKZ0009I: The application 50d712b0-768a-4ecb-9944-e457a6191166 has stopped successfully.
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.833 s - in application.rest.TNGTransactionRestEndpointIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

As you can see, the test application 50d712b0-768a-4ecb-9944-e457a6191166 is started, but then no tests actually get run before the test application is removed (I have logs in my tests that don't get printed to the console, and a test Asserts.assertTrue(false) that passes).

I've attached my sample project where you can replicate the issue. Just run mvn clean install.

liberty-runner-app.zip

[dipak-pawar]

@chyt: I tried to run mvn clean install, looks like something is wrong with liberty-maven-plugin configuration. I am getting exception while executing stop-server goal of liberty maven plugin. You can find a log here.

I tried with running mvn clean install on project provided by you. And I tried with updating version of liberty-maven-plugin to 2.1.2 and com.ibm.websphere.appserver.runtime:wlp-webProfile7 to 17.0.0.4. Getting the same result for both.

Can you give a try on your end and provide me working reproducer?

[dipak-pawar]

@chyt: I have resolved configuration issue for liberty-maven-plugin by adding following dependency for liberty-maven-plugin

<dependencies>
 <dependency>
  <groupId>org.apache.maven.wagon</groupId>
  <artifactId>wagon-http</artifactId>
  <version>2.8</version>
 </dependency>
</dependencies>

Now looking into the issue.

[chyt]

@dipak-pawar That's strange. I just ran the build again in a clean directory after deleting my Maven cache, and it is still working for me. Can you run mvn clean install -X and post the output? This will give us more verbose logs to identify the issue. Also what version of Maven are you using?

[dipak-pawar]

@chyt Don't worry about the issue which I was facing in the morning. I made it workable. If you still want it, I will give you :)

I found root cause of your failure. testng is not added inside container by arquillian as a part of arquillian-testng.jar which means testng classes are not available inside container.

For the time being to unblock you, just add testng as a library in your webarchive This will solve your issue.

@Deployment
    public static WebArchive createDeployment() {
        final PomEquippedResolveStage pomEquippedResolveStage = Maven.resolver().loadPomFromFile("pom.xml");
        File[] mavenFiles = pomEquippedResolveStage.importRuntimeDependencies().resolve()
                .withTransitivity().asFile();

        return ShrinkWrap.create(WebArchive.class).addPackage("application.rest").addPackage("application")
                .addAsLibraries(mavenFiles)
           .addAsLibraries(pomEquippedResolveStage.resolve("org.testng:testng").withTransitivity().asFile());
    }

I will submit PR for this in arquillian-core tomorrow, so you don't have to add testng explicitly as lib dependency which means arquillian will take care of it.