Test failing makes other tests running in parallel fail

[Lupiano]

Are you requesting a feature or reporting a bug?

Reporting a bug.

What is the current behavior?

When I am running my tests concurrently in more than one browser and one test fails, all the tests that are running at the same time also fail. The other tests continue its execution but the final report is this:

1. • Error in fixture.after hook -
     Unhandled promise rejection:

   [object Object]

What is the expected behavior?

Failure in one tests must not affect others.

How would you reproduce the current behavior (if this is a bug)?

Provide the test code and the tested page URL (if applicable)

Tested page URL:

Test code

Specify your

• operating system: Windows 10
• testcafe version: 0.22.0
• node.js version: 8.11.4

[Lupiano]

I think it is related to #2546 because with -u this bug disappears. Another detail is that the ones which fail with the "Unhandled promise rejection" error do not fail when executed isolated.

[AlexKamaev]

Hi, @Lupiano
You are right, the reason of the issues is in #2546. However, It is an expected and documented behavior. This mechanism is created to help developers to find uncaught errors and unhandled rejections, so if the issue occurs there is a possibility of error in the test code. The mechanism is implemented via Node's unhandledRejection event. When the event is handled, there is no reliable way to detect the place where error occurred, so we force TestCafe to fail all tests which are running at the moment.
This behavior is described in the documentation. Please refer the article http://devexpress.github.io/testcafe/documentation/using-testcafe/command-line-interface.html#-u---skip-uncaught-errors
If you are sure about your test code you can ignore this mechanism by using CLI --skip-uncaught-errors of -u option, as you've already mentioned.
 
I think we need to research why you have got the [object Object] string in the error message. Could please handle process.unhandledRejection event` in your test file?
 
Something like this:

var util = require('util');
process.on('unhandledRejection', (reason, p) => {
    console.log(util.format('%j', reason))
});

 

[Lupiano]

I'm getting this:

{"type":"externalAssertionLibraryError","isTestCafeError":true,"callsite":{"filename":"[filePath]","lineNum":[someLine],"callsiteFrameIdx":6,"stackFrames":[{},{},{},{},{},{},{},{},{},{},{},{},{}],"isV8Frames":true},"errMsg":"AssertionError: Some description: expected false to be truthy"}

It is a simple assertion error on some Selector.

[Lupiano]

BTW @AlexKamaev is there a way to skip these errors by code? Thanks for your time!

[AlexKamaev]

@Lupiano, thanks for your cooperation. The issue is unexpected, a simple assertion error should not cause unhandled rejection. Could I ask you to share your test file and test page or prepare a simple test, which shows the issue?
 
You can skip these errors only if you remove the listener from the process:

process.removeAllListeners('unhandledRejection');

Still, I do not recommend using this approach, because we need to detect the reason of the issue.
 

[Lupiano]

Hello @AlexKamaev, currently I cannot show you the test page, and the test file is a bit complex because uses some page objects, but it's something like this:

await t.click(reportsPage.pendingTab).expect(reportsPage.pendingNameInput.exists).ok('The "Name" input for the "Pending" table should exist').expect(...).ok(...)... and the chain continues.

This fails because the "reportsPage.pendingTab" selector is not possible to click (the element is under some other div that makes not possible to click it). The input element that is expected to appear does not, so there is an assertion error:

AssertionError: The "Name" input for the "Pending" table should
      exist: expected false to be truthy

This is what generates the Unhandled Rejection errors in the other tests running in parallel.

[AlexKamaev]

@Lupiano
I'll try to reproduce the issue. Until it is fixed, please use the '--skip-uncaught-errors' option.

[Lupiano]

Sure @AlexKamaev, if you need me to help you with something just tell me. Thanks!

[ymor]

Hi,
I've managed to recreate this heres an example:

import { Selector } from 'testcafe'; // first import testcafe selectors

fixture `Getting Started`// declare the fixture
    .page `https://devexpress.github.io/testcafe/example`;  // specify the start page

//then create a test and place your code there
test('My first test', async t => {

    const header = await Selector('header h1');
    const input = await Selector('#developer-name');

    await t
        .expect(header.textContent).eql('derp')
        .expect(input.textContent).eql('');
});

It only seems to occur when using chaining multiple expect asserts and when the failure does not occur in the last expect/assert. e.g if you flip the two expects around in the example, you won't see the unhandled promise rejection

BTW this error occurring did lead me to find another issue in my tests! I found some expects without asserts that were causing the test to incorrectly pass on previous versions of test cafe.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs or feature requests. For TestCafe API, usage and configuration inquiries, we recommend asking them on StackOverflow.