Execution time increased drastically after upgrading from 8.3.0 to 8.6.0

[byg0n3]

Current behavior

After upgrading cypress from 8.3.0 to 8.6.0, execution time of our tests (same amount of tests/suits) increased by ~10 min (almost doubled the initial time to execute).
Current execution results (version 8.6.0):

Desired behavior

We'd expect, that execution time stays the same or gets lower with cypress update, not increase almost two times.
Previous execution results (version 8.3.0):

Test code to reproduce

yarn upgrade [email protected]

Cypress Version

8.6.0

Other

Currently had to revert back to 8.3.0, because otherwise builds time out (due to the 30m aws-amplify build timeout setting).

[sgomezcnvdev]

I noticed the same issue migrating from 7.5.0 to 8.6.0

Cypress Version 7.5.0

Cypress Version 8.6.0

Cypress is running in Docker:
image: node12.18.3-chrome89-ff86

[CamilleDrapier]

In our case, we noticed that going from Cypress 8.5 to 8.6, incurred an average (over 6 runs for each version) suite-run time of:

8.5	8.6
158.6s	191.8s

Which is an increase of about 21% in the overall test time. Not sure what could be the source of this, but hopefully, this can be improved in the coming versions?

NB: Not sure if this is related, but we have also seen these warnings in our CI with the 8.6 version:

[453:1022/022105.354733:ERROR:bus.cc(392)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")

[lukeapage]

We also saw an increase from 8.5 - 8.6

would be interesting to hear if the first 2 reports are also better in 8.5.

[CommCody]

For us increases of about 50%+ in going from 8.4 to 8.6

[byg0n3]

We also saw an increase from 8.5 - 8.6

would be interesting to hear if the first 2 reports are also better in 8.5.

Our 8.5.0 results are ~ 8.3.0, so, most likely the issue is caused by 8.6.0.

[lukeapage]

I reviewed the commits between 8.5 and 8.6 and it’s most likely the electron upgrade, but I couldn’t see why that would cause this. I don’t have the time to investigate this in detail but if I did I would hit bisect using the master ci builds whilst they still exist to confirm which commit did it for sure

[rodrigo-vazquez]

In our case we tried today moving to Cypress 8.7 and we are seeing some tests taking nearly double the time especially when running them in GitHub Actions. Sticking to 8.5 for the moment. But we also understand it could be Electron 93 making it slower.

[DaniBedz]

Experiencing the same here when upgrading from 8.5 > 8.7.

Everything works fine when running cy:run locally.

I have noticed that tests that make several cy.request() calls at the beginning started failing completely, other tests were taking anywhere from 25% - 100% longer on GitHub Actions. Rolled back to 8.5 and all is fine again.

[szamanr]

i can confirm the issue is coming from electron. when i switch the browser to chrome via cypress.yaml, the issue disappears.

the video playback issue that i mentioned in #18740 also doesn't happen on chrome.

i'm running cypress in CI via cypress-io/github-action@v2. here's the config from cypress.yaml for switching a browser, in case it helps somebody:

- name: Run cypress tests
  uses: cypress-io/github-action@v2
  with:
    browser: chrome
    ...

[lszmit-nm]

We have observed the exact same problem moving from 8.3 to 8.6. Our test pipeline of 136 tests in 22 specs had its runtime escalate from ~30 minutes to over 2h (we're using Chrome, not Electron). Disabling video recording made a difference, but moving back to 8.3 had the most impact.

[petetnt]

More anecdotal evidence, with no code changes 8.7.0 performs 400% slower than 7.7.0 (which I updated to 8.7.0) and 8.5.0 (which I downgraded from 8.7.0). Running on GH Actions with multiple runs equaling around the same times from same setup (both on fresh cache and warmed cache).

[CamilleDrapier]

This slowness still seems to happen, for us, in the same proportion as described here in version 9.0.0

[AlexDaniel]

Same problem, downgrading to 8.5.0 helped a lot. Here are the timings:

EDIT: I have added 7.5.0 to comparison, but it seems like the timings are roughly the same as with 8.5.0.

Cypress 7.5.0	Cypress 8.5.0	Cypress 8.7.0

[szamanr]

is the team aware of this issue yet?

[helenaritson]

We are also experiencing the same issue - recent versions are much slower(both locally and in CI) and we are stuck on version 6.6.0
First biggest time difference was upgrading to 7.7.0 where test execution time jumped from ~37 to ~81 sec:

List of similar issues:

• Performance-Problems caused by requests to random domains #18057
• cy.visit() performance degradation after upgrade from 7.1 (persists in 8.3) #17759
• Tests run slower when upgrading from Cypress version 6.9.0 to 8.2.0 #17750
• Tests with many network requests are slower in 6.7+ since logging of all intercepts #18771

[AlexDaniel]

For what it is worth, I have tried 7.5.0 for our project (instead of 8.5.0) and I see no further performance improvement (compared to downgrading to 8.5.0 from 8.7.0). So, no, it's not the case of Wirth's law, and cypress is not getting slower the further we go, but there's clearly a performance regression in 8.6.0.

[BlueWinds]

is the team aware of this issue yet?

Sorry for the lack of response so far - performance regressions are something we take fairly seriously, and it's been causing us pain internally as well, due to much of our automation becoming flaky due to increased runtimes.

While we haven't pulled this ticket in specifically, it seems like at least some of this may be related to changes in how electron handles failed requests while under test automation control - @mjhenkes may have more information on that subject.

[jiangao1]

@helenaritson Thanks for showing the graph. I'm also holding the upgrade and in 8.3.1 now. Like the graph showed, we see a sizable increase from 8.3.1 to 8.7.0, and we had to roll back the upgrade. I will wait to see further performance data for later versions, and wait.

[jennifer-shehane]

Can anyone provide a reproducible example that we can run to reproduce the performance issue?

[CommCody]

I have no sample (yet), but I have tested our code base with the following versions (8.4 8.7 9.0) with different browsers (Electron Chrome Firefox). While I could discern some relevant differences between versions and browsers (runtime is between 6 and 8 minutes), none are as big as running this same code base on the az Pipelines ubuntu image on the cypress/included docker image (~50 minutes).

Not testable with version 9:
cypress-io/cypress-docker-images#555

I will try to distil a sample.

[jvavre]

I don't have exact comparison as others but noticed our CI runs going from 35mins in 6.x and 7.x to 50-55mins for 8.7.0 as for now with little to no changes in amount of TCs run.

[gguine-sweep]

hello, it's hard to get the time to create a reasonable reproduction example for something that global (demo stack, demo tests, benchmark).
Would a comparable set of full cypress log with DEBUG: cypress:* for same code base, context, test help ?

[jmoses]

Is it possible the real world app can be used as a reproduction case? I'm not sure if failing tests are a contributor (I feel like they are, but have not checked) or if the RWA has any that currently fail.

[tuomoa]

Our experiments shows that cypress 8.5.0+ uses a lot more cpu. We tested in docker container limited to 4 cores and 8gb of memory:

Version   Execution Time
9.1.1:    2:22:24
9.2.0:    2:09:50
8.5.0:    1:22:07

8.5.0 is also pretty similar in execution time even with 2 cores only, only a little bit slower.

However if we ran 8.5.0+ without docker limiting cpu usage, it took like 900% of cpu (the processes in total) and 9.1.1/9.2.0 took around 1h 30min. So it seems to me that the cpu usage is doubled/tripled and that is causing the slowing down if you have limited cpu resources.

[MaaikeFox]

I have the same problem. I switched to using chrome browser on CI runs and this brought running times back to normal so that I can upgrade Cypress

[szamanr]

still experiencing this issue with v9.3.1:

our current setup is v8.7 + chrome

• 8.7 + chrome 97 + ubuntu 20.04 ~= 11 min
• 9.3 + chrome 97 + ubuntu 20.04 ~= 11 min
• 9.3 + electron 94 + ubuntu 20.04 ~= 26 min

[FelipeLahti]

Experiencing the same issue here upgrading cypress from 8.4 to 9.4.1
E2E testing went from 3min to 6min
Component tests from 7min to 13min.

[tkovs]

Experiencing the same issue here upgrading cypress from 5.6.0 to 7.1.0. A complete run went from 30m to 1h.

[ChristianKlima]

Same with our solution. Time doubles with electron running on azure. With chrome cypress is a fast as it was before. Is it possible to fix something to run electron again or is it recommended to use chrome only?

[ChristianKlima]

Switch to chrome was not an option. Now we have a problem with same major Chrome versions check.
The Chrome browser is automatically updated on our test agents. Don't know why this is checked :(

You passed the --parallel flag, but we do not parallelize tests across different environments.

This machine is sending different environment parameters than the first machine that started this parallel run.

The existing run is:

In order to run in parallel mode each machine must send identical environment parameters such as:

• specs

• osName

• osVersion

• browserName

• browserVersion (major)

This machine sent the following parameters:

{

[lszmit-nm]

We have observed the exact same problem moving from 8.3 to 8.6. Our test pipeline of 136 tests in 22 specs had its runtime escalate from ~30 minutes to over 2h (we're using Chrome, not Electron). Disabling video recording made a difference, but moving back to 8.3 had the most impact.

More follow-up in terms of performance regressions since 8.3.
Our current pipeline runs on GitLab.com shared runners, and uses the node16.14.0-slim-chrome99-ff97 image as base.

We've managed to stabilise the runtime for our 28 specs (147 tests) around the 30 minute mark with the 9.x release by disabling videos, disabling test recording in dashboard, disabling parallel runs and switching from Electron to Chrome.

Today, as a test I've moved the entire setup to Cypress 10.0.1. Same base image, same specs, same config, just newer Cypress release. It (barely) managed to execute 12 specs before the 60 minute runtime limit killed the job. It took 16 minutes on a dev laptop.

The way I see it, something has dramatically changed in how Cypress behaves in CI/headless environments, running inside a docker container, since the 8.3 release. It's really becoming unusable at this point.

[csvan]

I understand it is not helpful, but I just want to echo that we are seeing the same degradation as described above on our (proprietary) suites, gradually getting worse from Cypress 7 to 9.7. Due to reports of even worse performance we are holding off migrating to 10 - unfortunately CI performance is almost unusable for us at this point.

My only observation is that this ONLY seems to affect CI/headless mode. Running cypress open still feels very fast and snappy, but as soon as we hit CI everything grinds to a halt.

If there is anything we can do to help dissect this issue, please let me know. I am not sure where to start looking.

[BlueWinds]

I understand it is not helpful, but I just want to echo that we are seeing the same degradation as described above on our (proprietary) suites, gradually getting worse from Cypress 7 to 9.7. Due to reports of even worse performance we are holding off migrating to 10 - unfortunately CI performance is almost unusable for us at this point.

My only observation is that this ONLY seems to affect CI/headless mode. Running cypress open still feels very fast and snappy, but as soon as we hit CI everything grinds to a halt.

If there is anything we can do to help dissect this issue, please let me know. I am not sure where to start looking.

We got some changes out in 10.3.0 which might fix your issue - no promise it's the same problem, but we've had one company report a 2x speedup in 10.3.0 compared to 10.2.0 (and 9.x).

[csvan]

@BlueWinds tested 9.7 vs 10.3.1 on one of our suites and saw an improvement of about 3 minutes (roughly 40% relative to average runtime under 9.x), awesome stuff!!

[BlueWinds]

We have some additional speedups in the pipeline. #22353 (comment) is about component testing, but the currently top candidate applies to e2e testing as well, cutting off about 300ms from loading every spec.

There's also V8 snapshots, which are a bit further out (and unclear exactly how much time they'll save - some, but too early to quantify yet).

The currently work on Detached DOM (#7306) should bring some performance improvement as well, though not on the scale of the other two tickets mentioned.

[BlueWinds]

We are also experiencing the same issue - recent versions are much slower(both locally and in CI) and we are stuck on version 6.6.0 First biggest time difference was upgrading to 7.7.0 where test execution time jumped from ~37 to ~81 sec:

That is a lovely benchmark and graphic, thank you for putting it together. If you could try the same under 10.3.1, it would be great to see how much further we still have to go to get back to the fast old days.

[Cellule]

Upgraded from 9.6.1 to 10.3.1 and test run time cut by about 3-4min, about 25% faster.
Amazing 🎉🎊

[bbartizek]

I saw that there were performance improvements mentioned in the changelog for 10.8.0, but running one of our existing test suites showed that this issue still is not resolved. Over the last 3 months on 8.1.0, the average runtime for the suite is 11m 44s, but my test on 10.8.0 took 30m 20s.

[BlueWinds]

@bbartizek - If you could provide us with a specific test which runs slower on 10.x, I'd love to look into it more.

I certainly believe it's happening, but in order to look into it, I first need to be able to isolate a case and examine it. I don't have access to your tests quite, so it running slower isn't much help in figuring out what the problem is. It's also very hard to compare across a gap of multiple major versions and a huge amount of code change, even if we have a specific test which does run slower on newer versions.

I understand the frustration, but unless you can help track down the problem, I'm not sure what else we can do beyond continuing to look at performance and making incremental improvements (which we are). 🤷‍♀️

[AlexDaniel]

@BlueWinds have you tried running any test suite on 8.3.0 vs 8.6.0 (and then vs the latest version)? What are your timings?

[BlueWinds]

I'm saying 8.3 vs 8.6 was a long time ago, and we (like the majority of projects) don't usually tend to support old versions.

We would welcome help identifying exactly what is now slow and improving it. But "still slow" is not useful or productive.

[jennifer-shehane]

Since this issue hasn't had activity in a while, we'll close the issue. We won't be investigating issues for 8.x releases and recommend upgrading to a later version of Cypress.

[AlexDaniel]

Has anybody done any measurements to confirm that the latest releases don't have this issue?

[lszmit-nm]

Has anybody done any measurements to confirm that the latest releases don't have this issue?

I know it's just a single datapoint, but we're currently running 226 tests in 38 specs averaging 4:00 minutes per run with 8-way parallelisation on 13.1. You can compare that to where we were just a year ago (a world of pain, in short).