Option to run tests in parallel

[AndreyBelym]

Are you requesting a feature or reporting a bug?

Feature

What is the current behavior?

TestCafe can run test only in series.

What is the expected behavior?

TestCafe can run tests concurrently.

[AndreyBelym]

Feature Proposal

Interface

Inspired by AVA

• Command line key -c/--concurrency <concurencyNumber> + API Runner method .concurrency(<concurencyNumber>)

testcafe chrome,firefox -c 2

runner
   .browsers('chrome,firefox')
   .concurrent(2)

• Store overall status of all tests that was ran on a connection and report it in BrowserProvider.reportJobResult(), so job result status will cover only tests runned on particular connection.

Implementation

• Should start <numberOfAutomatedBrowsers>*<concurencyNumber> browsers. Throw an error if number of remotes can't be divided by <concurrencyNumber>. When using remotes, and starting different kinds of browsers, the connection URL should be used <concurrencyNumber> times for each kind before starting the other kind.

• Current architecture allows different BrowserConnections (send the test URL to the browser) to share the same BrowserJob (generates test URLs). The only problem here is to filter .serial tests.

• Seems like fixture hook and reporter logic relies upon that tests are running in series. It needs to be reworked.

[inikulin]

As far as I remember reporter relies on serial execution as well.

[inikulin]

BTW, why do we need serial in the first place? What's the use case? Pure serial execution was never guaranteed by TestCase (same test runs in parallel in different browsers) and it worked well so far, so what itch we try to scratch here? Moreover, it will not work well with our reporting: since we have text-based reporters we can't break the order of tests. If you run "serial" tests first: you either don't report them immediately and wait for their fixture to start, or somehow split report in two parts: one for serial and one for parallel tests.

[AlexanderMoskovkin]

@AndreyBelym have updated the post, now it's

• .serial property for tests and fixtures to disallow running them concurrently. They should not change overall test execution order, just prevent next tests to be started.

test('1', () => );
test('2', () => ...);
test.serial('3', () => );
test('4', () => ...);
test('5', () => ...);
test.serial('6', () => );

runner.concurrent(2).run() =>

test1  test2
test3
test4  test5 
test6

It seems this approach doesn't break reporting.
In theory the concurrently feature can be useful when you run tests in a single browser and some tests can be run concurrently but some tests can't be (for example they change a database state). In this case you can mark several tests as serial and run others concurrently.

BUT, this option has some uncertainties.
For example if I run tests just in two browsers: testcafe chrome,firefox test.js, and I have a serial test. Should it be run in two browsers concurrently or it should be run one by one? How it should work if I run tests in two browsers with the concurrently option via testcafe chrome,firefox test.js -c 2 ?
So, do we really need the single test feature? /cc @DevExpress/testcafe

[inikulin]

In theory the concurrently feature can be useful when you run tests in a single browser and some tests can be run concurrently but some tests can't be (for example they change a database state).

Please, read carefully: we already have this behavior. Don't we try to solve a problem that doesn't exists here?

For example if I run tests just in two browsers: testcafe chrome,firefox test.js, and I have a serial test. Should it be run in two browsers concurrently or it should be run one by one? How it should work if I run tests in two browsers with the concurrently option via testcafe chrome,firefox test.js -c 2 ?

I have a great solution for this problem: don't add serial.

[inikulin]

...at least for now.

[Automation-Geek]

Please fo let us know the exact syntax to run or share sample code , very much appreciate if some one can.

[miherlosev]

See https://devexpress.github.io/testcafe/documentation/using-testcafe/common-concepts/concurrent-test-execution.html

[Automation-Geek]

It's not working for us, Tried severa time only serial execution is performing. I beleive there is some workaround to acheive this as they are opening in same port that could be run they are executing in serially.

[miherlosev]

@guptaakhil
It is difficult to understand the cause of the problem using the provided information .
Please, create a separate issue and specify operation system name, nodejs and testcafe versions.
In issue description telling us how you run the tests.

[Automation-Geek]

sure, we will be writing this into seperate thread.

[lock]

This thread has been automatically locked since it is closed and there has not been any recent activity. Please open a new issue for related bugs or feature requests. We recommend you ask TestCafe API, usage and configuration inquiries on StackOverflow.