Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running cucumber scenarios in parallel? #66

Open
Mulgish opened this issue Sep 6, 2022 · 3 comments
Open

Running cucumber scenarios in parallel? #66

Mulgish opened this issue Sep 6, 2022 · 3 comments

Comments

@Mulgish
Copy link

Mulgish commented Sep 6, 2022

Can you please advise if it is possible to run quarkus-cucumber extension tests in parallel mode?

When I try to setup Junit and Cucumber to run in parallel mode at scenario level, I am getting below errors:

@Local
Scenario: Test 2                               # src/test/resources/feature/Local.feature:10
  Given this scenario is running
  When I do something
  Then It should execute along other scenarios # com.mulgish.bdd.Steps.itShouldExecuteAlongOtherScenarios()
[ERROR] Tests run: 5, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 6.272 s <<< FAILURE! - in com.mulgish.bdd.CucumberTestIT
[ERROR] com.mulgish.bdd.CucumberTestIT.getTests  Time elapsed: 0.002 s  <<< ERROR!
java.lang.NullPointerException: Cannot invoke "java.time.temporal.Temporal.until(java.time.temporal.Temporal, java.time.temporal.TemporalUnit)" because "startInclusive" is null
	at java.base/java.time.Duration.between(Duration.java:490)
	at io.cucumber.core.runtime.CucumberExecutionContext.emitTestRunFinished(CucumberExecutionContext.java:109)
	at io.cucumber.core.runtime.CucumberExecutionContext.finishTestRun(CucumberExecutionContext.java:98)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$0(DynamicTestTestDescriptor.java:53)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptDynamicTest(QuarkusTestExtension.java:924)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:184)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$1(DynamicTestTestDescriptor.java:61)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptorCall.lambda$ofVoid$0(InvocationInterceptorChain.java:78)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:167)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:184)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$1(DynamicTestTestDescriptor.java:61)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptorCall.lambda$ofVoid$0(InvocationInterceptorChain.java:78)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.execute(DynamicTestTestDescriptor.java:60)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.execute(DynamicTestTestDescriptor.java:32)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

[ERROR] com.mulgish.bdd.CucumberTestIT.getTests  Time elapsed: 0.036 s  <<< ERROR!
io.cucumber.core.runner.DuplicateStepDefinitionException: Duplicate step definitions in com.mulgish.bdd.Steps.iDoSomething() and com.mulgish.bdd.Steps.iDoSomething()
	at io.cucumber.core.runner.CachingGlue.lambda$prepareGlue$3(CachingGlue.java:278)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at io.cucumber.core.runner.CachingGlue.prepareGlue(CachingGlue.java:272)
	at io.cucumber.core.runner.Runner.runPickle(Runner.java:72)
	at io.quarkiverse.cucumber.CucumberQuarkusTest.lambda$getTests$6(CucumberQuarkusTest.java:140)
	at io.cucumber.core.runtime.CucumberExecutionContext.lambda$runTestCase$5(CucumberExecutionContext.java:129)
	at io.cucumber.core.runtime.RethrowingThrowableCollector.executeAndThrow(RethrowingThrowableCollector.java:23)
	at io.cucumber.core.runtime.CucumberExecutionContext.runTestCase(CucumberExecutionContext.java:129)
	at io.quarkiverse.cucumber.CucumberQuarkusTest.lambda$getTests$7(CucumberQuarkusTest.java:140)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$0(DynamicTestTestDescriptor.java:53)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptDynamicTest(QuarkusTestExtension.java:924)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:184)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$1(DynamicTestTestDescriptor.java:61)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptorCall.lambda$ofVoid$0(InvocationInterceptorChain.java:78)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:167)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:184)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$1(DynamicTestTestDescriptor.java:61)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptorCall.lambda$ofVoid$0(InvocationInterceptorChain.java:78)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.execute(DynamicTestTestDescriptor.java:60)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.execute(DynamicTestTestDescriptor.java:32)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

[ERROR] com.mulgish.bdd.CucumberTestIT.getTests  Time elapsed: 0.079 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: failed in file:///home/sandbox/IdeaProjects/quarkus-cucumber-parallel-example/src/test/resources/feature/Local.feature at line 11
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:43)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:146)
	at io.quarkiverse.cucumber.CucumberQuarkusTest.lambda$getTests$7(CucumberQuarkusTest.java:145)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$0(DynamicTestTestDescriptor.java:53)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptDynamicTest(QuarkusTestExtension.java:924)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:184)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$1(DynamicTestTestDescriptor.java:61)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptorCall.lambda$ofVoid$0(InvocationInterceptorChain.java:78)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:167)
	at org.junit.jupiter.api.extension.InvocationInterceptor.interceptDynamicTest(InvocationInterceptor.java:184)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.lambda$execute$1(DynamicTestTestDescriptor.java:61)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptorCall.lambda$ofVoid$0(InvocationInterceptorChain.java:78)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.execute(DynamicTestTestDescriptor.java:60)
	at org.junit.jupiter.engine.descriptor.DynamicTestTestDescriptor.execute(DynamicTestTestDescriptor.java:32)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinTask.awaitDone(ForkJoinTask.java:436)
	at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:979)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.awaitFinished(NodeTestTask.java:236)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:158)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
[ERROR] Errors: 
[ERROR] com.mulgish.bdd.CucumberTestIT.getTests
[ERROR]   Run 1: CucumberTestIT.getTests » NullPointer Cannot invoke "java.time.temporal.Temporal.until(java.time.temporal.Temporal, java.time.temporal.TemporalUnit)" because "startInclusive" is null
[INFO]   Run 2: PASS
[INFO]   Run 3: PASS
[ERROR]   Run 4: CucumberTestIT>CucumberQuarkusTest.lambda$getTests$7:140->CucumberQuarkusTest.lambda$getTests$6:140 » DuplicateStepDefinition Duplicate step definitions in com.mulgish.bdd.Steps.iDoSomething() and com.mulgish.bdd.Steps.iDoSomething()
[ERROR]   Run 5: CucumberTestIT>CucumberQuarkusTest.lambda$getTests$7:145 failed in file:///home/sandbox/IdeaProjects/quarkus-cucumber-parallel-example/src/test/resources/feature/Local.feature at line 11

My code + configuration

CucumberTestIT.java
@CucumberOptions(features = "src/test/resources/feature/Local.feature", plugin = "json:target/cucumber-report/Report.json", tags="not @Ignore")
public class CucumberTestIT extends CucumberQuarkusTest   {
    public static void main(String[] args) {
        runMain(CucumberTestIT.class, args);
    }
}
Local.feature
@Local
Feature: Testing parallelism


  Scenario: Test 1
    Given this scenario is running
    When I do something
    Then It should execute along other scenarios

  Scenario: Test 2
    Given this scenario is running
    When I do something
    Then It should execute along other scenarios
Steps.java
public class Steps {

    private static final Logger LOGGER = LoggerFactory.getLogger(Steps.class);

    String scenarioName;

    @Before
    public void setup(Scenario scenario) {
        scenarioName = scenario.getName();
    }

    @Given("this scenario is running")
    public void thisScenarioIsRunning() {
        LOGGER.info("Running scenario " + scenarioName);
    }

    @When("I do something")
    public void iDoSomething() throws InterruptedException {
        Thread.sleep(1000);
    }

    @Then("It should execute along other scenarios")
    public void itShouldExecuteAlongOtherScenarios() {
        LOGGER.info("I hope {} scenario is running in parallel!", scenarioName);
    }

}
pom.xml
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0-M7</version>
    <configuration>
        <systemProperties>
            <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
        </systemProperties>
        <properties>
            <configurationParameters>
                cucumber.execution.parallel.enabled = true
                junit.jupiter.execution.parallel.enabled = true
                junit.jupiter.execution.parallel.mode.default = concurrent
                junit.jupiter.execution.parallel.mode.classes.default = same_thread
            </configurationParameters>
        </properties>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
            </goals>
        </execution>
    </executions>
</plugin>

If I remove parallel configuration under configurationParameters, everything works as expected but sequentially.

@Local
Scenario: Test 1                               # src/test/resources/feature/Local.feature:5
  Given this scenario is running               # com.mulgish.bdd.Steps.thisScenarioIsRunning()
2022-09-06 23:47:58,875 INFO  [com.lgi.bdd.Steps] (main) Running scenario Test 1
  When I do something                          # com.mulgish.bdd.Steps.iDoSomething()
  Then It should execute along other scenarios # com.mulgish.bdd.Steps.itShouldExecuteAlongOtherScenarios()
2022-09-06 23:47:59,887 INFO  [com.lgi.bdd.Steps] (main) I hope Test 1 scenario is running in parallel!

@Local
Scenario: Test 2                               # src/test/resources/feature/Local.feature:10
  Given this scenario is running               # com.mulgish.bdd.Steps.thisScenarioIsRunning()
2022-09-06 23:47:59,892 INFO  [com.lgi.bdd.Steps] (main) Running scenario Test 2
  When I do something                          # com.mulgish.bdd.Steps.iDoSomething()
  Then It should execute along other scenarios # com.mulgish.bdd.Steps.itShouldExecuteAlongOtherScenarios()
2022-09-06 23:48:00,897 INFO  [com.lgi.bdd.Steps] (main) I hope Test 2 scenario is running in parallel!

2 Scenarios (2 passed)
6 Steps (6 passed)
0m2.067s

Process finished with exit code 0
@msnijder30
Copy link

@Mulgish I was wondering if you ever got this working? I am also trying out all sorts of things like this, but just cannot get it to run in parallel. @gsmet Maybe you are aware if should work or not?

@Mulgish
Copy link
Author

Mulgish commented Nov 1, 2023

@msnijder30 It is not possible to do unless the plugin is adapted to support this. If you use plain cucumber (without Quarkus spinning up), it works.

@msnijder30
Copy link

@Mulgish Thanks a lot for the quick reply, unfortunate that it's not possible :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@msnijder30 @Mulgish