TestCafe performance issues - Slow page loads, High CPU usage, Browser disconnect issues etc.

[sijosyn]

What is your Test Scenario?

Download the provided test repo and execute the tests. The test scripts basically checks the functionality of the given website.
I'm currently in the process of migrating our projects Selenium tests into TestCafe and I'm halfway through the process. When I tried to execute the migrated tests in different browsers (both on Mac & Win), I noticed several issues as mentioned below.

What is the Current behavior?

Multiple Issues:
1 - The page is very slow to load when run via TestCafe. If I manually load the page in the browser (with cache & cookie cleared), the page loads reasonably fast. So I'm not sure what's causing the page to load slow inside the TestCafe server. Try to run the tests in Firefox (on Mac or Win) and you can easily observe this issue. We have longer test execution time because of this issue.
2 - The CPU usage also goes 100% ('Google Chrome Helper' & 'FirefoxCP Web Content' processes) when executing the test suite in Chrome & Firefox on Mac. You can hear loud fan noice when this happens. (see screenshot below)
3 - Test scripts keep failing in Firefox (on Mac & Win) & Edge on Win. I'm unable to execute the test suite. I keep getting the browser disconnected error. (see screenshot below)
4 - Also test execution fails in Chrome on Windows. Getting 'write ECONNABORTED' in console (see screenshot below)

What is the Expected behavior?

None of the above issues should happen.

What is your web application and your TestCafe test code?

Download this repo - https://github.com/sijosyn/testcafe-syntests and run the test suite.

Your complete test code (or attach your test files):

Above provided repo contains all the test code.

Your complete test report:

Screenshots:

---

Firefox browser disconnect error:

---

Chrome disconnect error (on Win):

---

Edge browser disconnect error

---

---

Steps to Reproduce:

Please read the README provided in the repo - https://github.com/sijosyn/testcafe-syntests

Your Environment details:

• testcafe version: 0.23.0
• node.js version: 8.9.4
• command-line arguments: testcafe firefox desktop -e
• browser name and version: Firefox 63, Safari 12, Chrome 70, Edge 17
• platform and version: macOS 10.13, Win 10
• other:

[miherlosev]

The page is very slow to load when run via TestCafe.

We will investigate it more deeply. I will notify you of our results later.

The CPU usage also goes 100%

I cannot reproduce this behavior. On my machine I have CPU usage about 60 %. Google Chrome takes about 20%. It's a normal behavior because TestCafe emulates user actions very fast.

Test scripts keep failing in Firefox (on Mac & Win) & Edge on Win

I cannot reproduce it.

Also test execution fails in Chrome on Windows. Getting 'write ECONNABORTED' in a console (see screenshot below)

There is a separate issue for this - DevExpress/testcafe-hammerhead#1744.
We will fix it during the current development sprint.

[sijosyn]

@miherlosev Thanks for your prompt response and investigating this issue.

I cannot reproduce this behavior. On my machine I have CPU usage about 60 %. Google Chrome takes about 20%. It's a normal behavior because TestCafe emulates user actions very fast.

I've observed the CPU usage goes up while its waiting to finish loading the page and before it starts to interact on the page. This issue happens on Mac for me. Could you please check this issue on Mac if possible?. My system info is provided below.

Test scripts keep failing in Firefox (on Mac & Win) & Edge on Win

If possible, could you please try to run the test suite in following browser & OS combination and see if you can reproduce the browser disconnect issue:
Firefox 63.0.0 / Mac OS X 10.13.0
Firefox 63.0.0 / Win 10.0.0
Edge 16.16299 / Win 10.0.0

I would like to see the 'Browser disconnect' issue resolved as its one of the top annoying issue for me. Its been around for a while for now and haven't been addressed yet. I understand its hard to reproduce the browser disconnect issue but please see if the team can find the root cause of this issue and fix it.

[lukastaegert]

I can confirm similar performance issues for
Safari 12.0.0 / Mac OS X 10.14.0

A simple hello world test took 1m 19s where most of this time was spent navigating to the initial page. Other browsers were noticeably faster:
Firefox 62.0.0 / Mac OS X 10.14.0: 12s
Chrome 70.0.3538 / Mac OS X 10.14.0: 4s

Note that Firefox is still quite a bit slower in loading the page. Safari is basically unusable.

[lukastaegert]

Did some more testing: I only observe this when testing on localhost where I used webpack dev server to serve my page. Remote pages seem to be fast.

[ZeddEn]

I started having this problem starting from version 0.22.0 (now using 0.23.1). Before it used to hang the build for hours but with a fix introduced on the 0.22.0 now, it just disconnects the browser(s). This may be strongly linked to running tests on multi-browsers (chrome & firefox).

I am using TeamCity to run the builds with the following configuration:

• UserAgents: Chrome 70.0.3538 / Windows 10.0.0,Firefox 61.0.0 / Windows 10.0.0
• Testcafe version: 0.23.1
• Node version: 8.11.3

There is the error message I get on my CI output:

Error: Unable to establish one or more of the specified browser connections. This can be caused by network issues or remote device failure. at C:\BuildAgent\work\4a74606c58d6f60f\node_modules\testcafe\src\runner\browser-set.js:84:30 at next (native) at step (C:\BuildAgent\work\4a74606c58d6f60f\node_modules\babel-runtime\helpers\asyncToGenerator.js:17:30) at C:\BuildAgent\work\4a74606c58d6f60f\node_modules\babel-runtime\helpers\asyncToGenerator.js:35:14 at Promise.F (C:\BuildAgent\work\4a74606c58d6f60f\node_modules\core-js\library\modules\_export.js:36:28) at C:\BuildAgent\work\4a74606c58d6f60f\node_modules\babel-runtime\helpers\asyncToGenerator.js:14:12 at BrowserSet._waitConnectionsOpened (C:\BuildAgent\work\4a74606c58d6f60f\node_modules\testcafe\src\runner\browser-set.js:77:37) at _waitConnectionsOpened (C:\BuildAgent\work\4a74606c58d6f60f\node_modules\testcafe\src\runner\browser-set.js:107:35) at invokeCallback (C:\BuildAgent\work\4a74606c58d6f60f\node_modules\pinkie\index.js:60:12) at Array.forEach (native) Process exited with code 1

I am also using this on my runner:
.then(failedCount => { console.log('Tests failed: ' + failedCount); testcafe.close(); process.exit(failedCount ? 1 : 0); }
I think the problem is related to firefox as when I disactivated multi-browser (removing firefox), I have not seen the problem so far.

[miherlosev]

@sijosyn

Also test execution fails in Chrome on Windows. Getting 'write ECONNABORTED' in a console (see screenshot below)

We've fixed this problem in [email protected]. Try to run your tests with this version and let us know if the other issues from your list above are reproduced.

@lukastaegert

A simple hello world test took 1m 19s where most of this time was spent navigating to the initial page.

It's a very strange. Could you please provide a simple example so that we will reproduce it on our side?

[lukastaegert]

Admittedly only the test was "Hello World", the application under test was not. So I tried to create another repo using generic code but was not able to. In my original project I managed to track it down to my usage of @material-ui/icons. Once I removed the single used icon, Safari was immediately fast. So I set up a new project just using this icon but this was fast as well.
One difference was, though, that the new project was a create-react-app JavaScript project while the original one was a TypeScript project. Not sure how long I will try to find a simple reproduction. As for me, as I can live without those icons I am fine, so you may just take this as a hint that there may be something fishy going on. Especially considering that the slowness was 100% reproducible and 100% linked to the import (but usage! The component in question was never displayed in the initial view) of these icons at least in a TS setup and that it only occurred on Safari.

[sijosyn]

@miherlosev

We've fixed this problem in [email protected]. Try to run your tests with this version and let us know if the other issues from your list above are reproduced.

The 'write ECONNABORTED' issue is resolved in [email protected]. However, the other performance issues I reported (Slow page loads, High CPU usage, Browser disconnect issues) still exists in [email protected].

[sijosyn]

Issue is too old now and can't reproduce. Closing.

[lock]

This thread has been automatically locked since it is closed and there has not been any recent activity. Please open a new issue for related bugs or feature requests. We recommend you ask TestCafe API, usage and configuration inquiries on StackOverflow.