Write Customization tests in python #2918

hubertdeng123 · 2024-03-21T23:55:22Z

This PR adds a few things

Codecov reporting for self-hosted
Proper flaky test detection for all of integration tests in self-hosted with pytest-sentry
All integration tests are now running through pytest

codecov · 2024-03-22T17:48:16Z

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

hubertdeng123 · 2024-03-22T22:23:11Z

Flaky test reporting works!
https://sentry.sentry.io/issues/5093043103/?project=6627632&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=0&utc=true

hubertdeng123 · 2024-03-25T21:38:59Z

.github/workflows/test.yml

@@ -16,7 +16,7 @@ defaults:
 jobs:
  e2e-test:
    if: github.repository_owner == 'getsentry'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04


bumped this while at it

nit: Imo we should split this into a separate PR, since I could imagine cases where we want to only roll this bump back.

hubertdeng123 · 2024-03-25T21:39:13Z

.github/workflows/test.yml

          ### pytest-sentry configuration ###
          if [ "$GITHUB_REPOSITORY" = "getsentry/self-hosted" ]; then
-            echo "PYTEST_SENTRY_DSN=${{ env.SELF_HOSTED_TESTING_DSN }}" >> $GITHUB_ENV
+            echo "PYTEST_SENTRY_DSN=$SELF_HOSTED_TESTING_DSN" >> $GITHUB_ENV


This was previously not being picked up correctly

hubertdeng123 · 2024-03-25T21:42:11Z

_integration-test/conftest.py

            time.sleep(1)
        else:
            if response.status_code == 200:
                break
    else:
        raise AssertionError("timeout waiting for self-hosted to come up")

+    if request.config.getoption("--customizations") == "True":
+        os.environ['TEST_CUSTOMIZATIONS'] = "True"
+        script_content = '''\


These lines are definitely covered by testing, not sure why Codecov is complaining here.

That's the reason why Test / integration test (True, v2.0.1) and Test / integration test (True, v2.7.0) are taking 8 minutes longer

nit: It's pretty hard for a naive observer to figure out what the "True" in these variants means. Maybe we could have values be Customization Enabled vs Customization Disabled rather than True/False?

Apparently, you can create a custom name for the job. Now it looks like
Test / integration test v2.0.1 - customizations disabled

azaslavsky · 2024-03-25T23:13:17Z

.github/workflows/test.yml

@@ -16,7 +16,7 @@ defaults:
 jobs:
  e2e-test:
    if: github.repository_owner == 'getsentry'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04


nit: Imo we should split this into a separate PR, since I could imagine cases where we want to only roll this bump back.

azaslavsky · 2024-03-25T23:15:22Z

_integration-test/conftest.py

            time.sleep(1)
        else:
            if response.status_code == 200:
                break
    else:
        raise AssertionError("timeout waiting for self-hosted to come up")

+    if request.config.getoption("--customizations") == "True":
+        os.environ['TEST_CUSTOMIZATIONS'] = "True"
+        script_content = '''\


nit: It's pretty hard for a naive observer to figure out what the "True" in these variants means. Maybe we could have values be Customization Enabled vs Customization Disabled rather than True/False?

azaslavsky · 2024-03-25T23:17:31Z

_integration-test/test_run.py

+    for command in commands:
+        result = subprocess.run(command, check=False)
+        if os.getenv("TEST_CUSTOMIZATIONS", "False") == "True":
+            assert result.returncode == 0


Does test_customization run in both matrix variants?

Yes, it does. If customizations enabled, it checks to ensure that they are truly enabled by making sure a dependency can be imported that is included in the custom image (ldap). If not, it checks to ensure that an error is thrown when trying to import a dependency that comes from the custom image.

hubertdeng123 added 6 commits March 21, 2024 16:54

port everything to python

981afa1

use autouse fixture

23bade6

fix precommit

fa0bcaa

fix some syntax issues

ac67a3b

reinstall self-hosted after making customizations

fd28ad4

add codecov

68da8b0

hubertdeng123 mentioned this pull request Mar 22, 2024

Clean up flakey CI getsentry/team-ospo#242

Closed

6 tasks

track flaky test failures

5c2fd16

hubertdeng123 force-pushed the hubertdeng123/customization-tests-python branch from 11f9bea to 5c2fd16 Compare March 22, 2024 22:19

only enable flaky tests for master

2ccd1a5

hubertdeng123 force-pushed the hubertdeng123/customization-tests-python branch from c93b5e5 to 2ccd1a5 Compare March 22, 2024 22:43

catch exception for general httpx networkerror

82eed1e

hubertdeng123 force-pushed the hubertdeng123/customization-tests-python branch from bcdccaa to 82eed1e Compare March 25, 2024 16:30

hubertdeng123 added 2 commits March 25, 2024 10:28

add test customizations

fbe3713

customization tests working finally?

1799f09

hubertdeng123 force-pushed the hubertdeng123/customization-tests-python branch from a4f67e4 to 1799f09 Compare March 25, 2024 19:30

hubertdeng123 added 3 commits March 25, 2024 12:31

add newline to eof

867ed00

fix customizations tests

fb6d766

add -c flag

fe79de0

hubertdeng123 marked this pull request as ready for review March 25, 2024 21:38

hubertdeng123 requested a review from azaslavsky March 25, 2024 21:38

hubertdeng123 commented Mar 25, 2024

View reviewed changes

remove no longer needed shell files

b61dd79

azaslavsky approved these changes Mar 25, 2024

View reviewed changes

hubertdeng123 added 2 commits March 26, 2024 09:20

try and set custom name for integration test jobs

6732677

add customizations enabled

a6fa7b0

use retries on installing self hosted

2594145

hubertdeng123 merged commit 9b3b9bc into master Mar 26, 2024
12 checks passed

hubertdeng123 deleted the hubertdeng123/customization-tests-python branch March 26, 2024 18:14

hubertdeng123 mentioned this pull request Mar 26, 2024

Add flaky test detector #2874

Closed

github-actions bot locked and limited conversation to collaborators Apr 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write Customization tests in python #2918

Write Customization tests in python #2918

hubertdeng123 commented Mar 21, 2024 •

edited

Loading

codecov bot commented Mar 22, 2024 •

edited

Loading

hubertdeng123 commented Mar 22, 2024

hubertdeng123 Mar 25, 2024

azaslavsky Mar 25, 2024

hubertdeng123 Mar 25, 2024

hubertdeng123 Mar 25, 2024

hubertdeng123 Mar 25, 2024

azaslavsky Mar 25, 2024

hubertdeng123 Mar 26, 2024

azaslavsky Mar 25, 2024

azaslavsky Mar 25, 2024

azaslavsky Mar 25, 2024

hubertdeng123 Mar 26, 2024

Write Customization tests in python #2918

Write Customization tests in python #2918

Conversation

hubertdeng123 commented Mar 21, 2024 • edited Loading

codecov bot commented Mar 22, 2024 • edited Loading

Welcome to Codecov 🎉

hubertdeng123 commented Mar 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hubertdeng123 commented Mar 21, 2024 •

edited

Loading

codecov bot commented Mar 22, 2024 •

edited

Loading