[#1570] Speed up CI #690

pi-sigma · 2023-06-22T17:06:17Z

Various tweaks to speed up the tests:

run tests in parallel
skip old data migration tests
use MD5 for password hashing
optimize postgres
disable logging for WARNING and below

pi-sigma · 2023-06-22T20:16:34Z

There are two issues with running our tests in parallel:

Elasticsearch uses persistent connections for their search client which doesn't play well with multiprocessing (see here). Unfortunately, there is no public API to tweak things. One option would be to mess with the internals to ensure that each Django test worker is coupled with its own Elastic client. I've looked into this but decided not to pursue this avenue. The solution is bound to be messy and will likely break at some point. The second option was to split the test suite into two, run the "ordinary" tests in parallel, and do a second run for the tests that resist being run in parallel.
Our playwright tests don't play nice with multiprocessing either as Django can't serialize the relevant test classes. For now I simply lumped them together with the Elastic test cases, but we should look into the source of this eventually.

codecov-commenter · 2023-06-23T10:35:21Z

Codecov Report

Merging #690 (b9654a7) into develop (a0506b0) will decrease coverage by 2.74%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #690      +/-   ##
===========================================
- Coverage    96.31%   93.58%   -2.74%     
===========================================
  Files          699      697       -2     
  Lines        24670    24595      -75     
===========================================
- Hits         23762    23016     -746     
- Misses         908     1579     +671

Files Changed	Coverage Δ
...c/open_inwoner/accounts/tests/test_auth_2fa_sms.py	`100.00% <ø> (ø)`
src/open_inwoner/conf/ci.py	`100.00% <100.00%> (ø)`
src/open_inwoner/haalcentraal/tests/test_signal.py	`100.00% <100.00%> (ø)`
src/open_inwoner/plans/tests/test_views.py	`75.78% <100.00%> (-24.22%)`	⬇️
src/open_inwoner/search/tests/test_feedback.py	`17.60% <100.00%> (-82.41%)`	⬇️
src/open_inwoner/search/tests/test_logging.py	`51.85% <100.00%> (-48.15%)`	⬇️
src/open_inwoner/search/tests/test_page.py	`19.73% <100.00%> (-80.27%)`	⬇️
...n_inwoner/search/tests/test_search_autocomplete.py	`20.68% <100.00%> (-79.32%)`	⬇️
.../open_inwoner/search/tests/test_search_products.py	`25.80% <100.00%> (-74.20%)`	⬇️

... and 19 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Bartvaderkin · 2023-08-21T09:31:33Z

src/open_inwoner/conf/test.py

+if len(sys.argv) > 1 and sys.argv[1] == "test":
+    logging.disable(logging.CRITICAL)



I'm not sure if this is what we want. Supposedly we still want to see something when a real problem occurs?

I changed it so we keep logs for level >= ERROR

src/open_inwoner/accounts/tests/test_actions_expire.py

.coverage.fedora.1089256.271783

alextreme · 2023-08-28T12:31:14Z

src/open_inwoner/pdc/tests/test_data_migrations.py

@@ -1,3 +1,5 @@
+from unittest import skip
+


I'd recomment to remove instead of skip these tests if they're no longer necessary

src/open_inwoner/search/tests/test_search_products_boost.py

Bartvaderkin · 2023-09-04T10:31:20Z

src/open_inwoner/pdc/tests/test_data_migrations.py

@@ -1,106 +0,0 @@
-from django.contrib.gis.geos import Point


Was there a specific reason we had to let these go?

If the amount of migrations is an issue we could look into squashing the migrations. IIRC it also had some complications with data migrations.

Bartvaderkin · 2023-09-04T10:40:06Z

src/open_inwoner/conf/ci.py

@@ -13,7 +13,7 @@
    {
        "django": {
            "handlers": ["django"],
-            "level": "WARNING",
+            "level": "ERROR",


I'm having doubts about the logging levels.

Maybe we should move that to another issue en get the config changes and parallel test merged.

turn off fsync for Postgres update actions in workflow

- the functionality will be re-implemented later on via a test matrix in GH actions

src/open_inwoner/conf/test.py

pi-sigma force-pushed the refactor-tests branch from fafd995 to 7ae603e Compare June 22, 2023 17:30

pi-sigma changed the title ~~run tests in parallel~~ refactor tests Jun 22, 2023

pi-sigma force-pushed the refactor-tests branch from befe57a to b1f1f1a Compare June 23, 2023 10:06

pi-sigma changed the title ~~refactor tests~~ [#1570] Speed up CI Jun 23, 2023

pi-sigma force-pushed the refactor-tests branch 4 times, most recently from 8120eca to e9a4692 Compare June 26, 2023 09:00

pi-sigma force-pushed the refactor-tests branch 8 times, most recently from 6272330 to 29cd3f9 Compare August 11, 2023 07:46

pi-sigma marked this pull request as ready for review August 11, 2023 08:29

pi-sigma requested a review from Bartvaderkin August 11, 2023 08:29

pi-sigma force-pushed the refactor-tests branch 4 times, most recently from 95697a0 to 2b7ea87 Compare August 17, 2023 11:02

Bartvaderkin suggested changes Aug 21, 2023

View reviewed changes

alextreme reviewed Aug 28, 2023

View reviewed changes

.coverage.fedora.1089256.271783 Outdated Show resolved Hide resolved

alextreme reviewed Aug 28, 2023

View reviewed changes

pi-sigma force-pushed the refactor-tests branch from 2b7ea87 to a504070 Compare August 28, 2023 14:12

pi-sigma requested a review from Bartvaderkin August 28, 2023 14:34

pi-sigma force-pushed the refactor-tests branch from a504070 to b141004 Compare August 29, 2023 13:05

pi-sigma force-pushed the refactor-tests branch 2 times, most recently from 8c3987a to e0e5870 Compare August 29, 2023 13:40

Bartvaderkin suggested changes Sep 4, 2023

View reviewed changes

pi-sigma added 6 commits September 11, 2023 09:15

[#1570] run tests in parallel

e922cc4

[#1570] skip data migration tests

5d91aeb

[#1570] use MD5 for password hashing

8f18aa5

turn off fsync for Postgres update actions in workflow

[#1570] remove multi-browser support for playwright tests

59ef6ba

- the functionality will be re-implemented later on via a test matrix in GH actions

[#1570] use custom GH action to set up backend environment

9665baa

[#1570] create separate ci job for e2e tests

35e3680

pi-sigma force-pushed the refactor-tests branch from e0e5870 to b9654a7 Compare September 11, 2023 07:16

pi-sigma requested a review from Bartvaderkin September 11, 2023 07:39

Bartvaderkin suggested changes Sep 11, 2023

View reviewed changes

src/open_inwoner/conf/test.py Outdated Show resolved Hide resolved

[#1570] disable logging except for errors and critical messages

99046b3

pi-sigma force-pushed the refactor-tests branch from b9654a7 to 99046b3 Compare September 11, 2023 10:33

Bartvaderkin approved these changes Sep 11, 2023

View reviewed changes

alextreme merged commit defa012 into develop Sep 11, 2023
14 checks passed

alextreme deleted the refactor-tests branch September 11, 2023 12:05

swrichards mentioned this pull request Jun 26, 2024

[#2589] Run CI tests in parallel #1282

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#1570] Speed up CI #690

[#1570] Speed up CI #690

pi-sigma commented Jun 22, 2023 •

edited

Loading

pi-sigma commented Jun 22, 2023 •

edited

Loading

codecov-commenter commented Jun 23, 2023 •

edited

Loading

Bartvaderkin Aug 21, 2023

pi-sigma Aug 28, 2023

alextreme Aug 28, 2023

Bartvaderkin Sep 4, 2023

Bartvaderkin Sep 4, 2023

alextreme Sep 4, 2023

		if len(sys.argv) > 1 and sys.argv[1] == "test":
		logging.disable(logging.CRITICAL)

[#1570] Speed up CI #690

[#1570] Speed up CI #690

Conversation

pi-sigma commented Jun 22, 2023 • edited Loading

pi-sigma commented Jun 22, 2023 • edited Loading

codecov-commenter commented Jun 23, 2023 • edited Loading

Codecov Report

Bartvaderkin Aug 21, 2023

Choose a reason for hiding this comment

pi-sigma Aug 28, 2023

Choose a reason for hiding this comment

alextreme Aug 28, 2023

Choose a reason for hiding this comment

Bartvaderkin Sep 4, 2023

Choose a reason for hiding this comment

Bartvaderkin Sep 4, 2023

Choose a reason for hiding this comment

alextreme Sep 4, 2023

Choose a reason for hiding this comment

pi-sigma commented Jun 22, 2023 •

edited

Loading

pi-sigma commented Jun 22, 2023 •

edited

Loading

codecov-commenter commented Jun 23, 2023 •

edited

Loading