Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix flaky test on tests::image::test_utils.py::ReleaseUtilsTest::test_get_docker_client #411

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

lonly7star
Copy link

@lonly7star lonly7star commented Nov 20, 2021

This PR aims to fix the flaky test on tests::image::test_utils.py::ReleaseUtilsTest::test_get_docker_client so the test could pass single run, multiple test run, and random test run.

The result

The test will fail on the multipe re-runs with pytest flake-finder, the following error will raised start on the second test run
> client_mock.assert_called_with( base_url=None, tls=None, version=DEFAULT_DOCKER_API_VERSION)
E AssertionError: expected call not found.
E Expected: mock(base_url=None, tls=None, version='1.21')
E Actual: mock(base_url='http://127.0.0.1', version='1.40', tls=<Mock name='mock.tls.TLSConfig()' id='140319742926704'>)

Steps to reproduce the issue

  1. install pytest flaky finder with pip install pytest-flakefinder
  2. run pytest with flake-finder with command pytest -k tests/image/test_utils.py --flake-finder

Issue of the code

The reason is that the test assumes the os.environ variables DOCKER_HOST, DOCKER_TLS_VERIFY, and DOCKER_API_VERSION are None at the beginning of the testing method. However, those variables will be initialized in the later part of the test and didn't get removed. Therefore started on the second test run the test will fail due to different code flow on `utils.opy::get_docker_client'

Proposed solution

Adding three if statement to delete those environmental variables if they are not None, so the test function stay fresh in each run.

fix flaky test on test_get_docker_client::ReleaseUtilsTest::test_get_docker_client
@lonly7star lonly7star changed the title fix flaky test on tests::image::test_utils.py::test_get_docker_client::ReleaseUtilsTest::test_get_docker_client fix flaky test on tests::image::test_utils.py::ReleaseUtilsTest::test_get_docker_client Nov 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant