Integration tests in python #2892
Conversation
hubertdeng123
force-pushed
the
hubertdeng123/integration-tests-python
branch
from
a824e7d
to
1eadfea
Compare
| @@ -15,12 +15,11 @@ export MINIMIZE_DOWNTIME=0 | |||
|
|
|||
| if [[ "$test_option" == "--initial-install" ]]; then | |||
There was a problem hiding this comment.
The goal is to rewrite this file entirely in python using pytest, but baby steps!
_integration-test/run.py
Outdated
| cls.session = requests.Session() | ||
| response = None | ||
| run_shell_command("docker compose --ansi never up -d") | ||
| for i in range(TIMEOUT_SECONDS): |
There was a problem hiding this comment.
this is pretty ugly, open to suggestions on how to make polling for a response cleaner
hubertdeng123
force-pushed
the
hubertdeng123/integration-tests-python
branch
from
03d714b
to
bf0905b
Compare
_integration-test/run.py
Outdated
| import subprocess | ||
| import os | ||
| from functools import lru_cache | ||
| import requests |
There was a problem hiding this comment.
I would not recommend using requests in new code
if you can get away with it, stdlib urllib.request does not require additional dependencies. httpx is a modern replacement if you find urllib.request too cumbersome
There was a problem hiding this comment.
switched to using httpx, since I wanted a good way to save cookies
There was a problem hiding this comment.
What's the reason why requests is not the preferred choice?
| response = client.get( | ||
| request, follow_redirects=True, headers={"Referer": SENTRY_TEST_HOST} | ||
| ) | ||
| if response.status_code == 200: |
There was a problem hiding this comment.
This function won't work for spans as it's waiting for a span to be ingested to succeed, so it's receiving a 200 no matter what and looking for the data field to not be empty.
We risk flaky tests if we leave it at that. Maybe we can have a custom behavior via a function to return True/False if it's considered a success of if we need to retry?
There was a problem hiding this comment.
Ah, thanks for the context. I added a optional validator function passed into poll_for_response that checks the response json to check to ensure the data field length is greater than 0
| @@ -15,7 +15,7 @@ export MINIMIZE_DOWNTIME=0 | |||
|
|
|||
| if [[ "$test_option" == "--initial-install" ]]; then | |||
| echo "Testing initial install" | |||
| source _integration-test/run.sh | |||
| pytest --reruns 5 _integration-test/run.py | |||
There was a problem hiding this comment.
Is that even needed? If we have proper retries when polling data, we likely don't need to retry the tests up to 5 times, do we?
| @@ -15,7 +15,7 @@ export MINIMIZE_DOWNTIME=0 | |||
|
|
|||
| if [[ "$test_option" == "--initial-install" ]]; then | |||
| echo "Testing initial install" | |||
| source _integration-test/run.sh | |||
| pytest --reruns 5 _integration-test/run.py | |||
There was a problem hiding this comment.
I would recommend not having reruns -- they're a crutch that allows flakiness to creep in and ideally we wouldn't have them in sentry
There was a problem hiding this comment.
Primary reason behind this is because docker compose itself is quite flaky in CI, and I wanted to have a way to rerun the fixture that brings self-hosted up and report the flakiness
hubertdeng123
force-pushed
the
hubertdeng123/integration-tests-python
branch
from
935a84b
to
d20b057
Compare
This adds a couple of exciting things and will be the first step into modernizing the test suite for self-hosted.