fix flaky test on tests::image::test_utils.py::ReleaseUtilsTest::test_get_docker_client #411
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to fix the flaky test on tests::image::test_utils.py::ReleaseUtilsTest::test_get_docker_client so the test could pass single run, multiple test run, and random test run.
The result
The test will fail on the multipe re-runs with pytest flake-finder, the following error will raised start on the second test run
> client_mock.assert_called_with( base_url=None, tls=None, version=DEFAULT_DOCKER_API_VERSION)
E AssertionError: expected call not found.
E Expected: mock(base_url=None, tls=None, version='1.21')
E Actual: mock(base_url='http://127.0.0.1', version='1.40', tls=<Mock name='mock.tls.TLSConfig()' id='140319742926704'>)
Steps to reproduce the issue
pip install pytest-flakefinder
pytest -k tests/image/test_utils.py --flake-finder
Issue of the code
The reason is that the test assumes the os.environ variables
DOCKER_HOST
,DOCKER_TLS_VERIFY
, andDOCKER_API_VERSION
are None at the beginning of the testing method. However, those variables will be initialized in the later part of the test and didn't get removed. Therefore started on the second test run the test will fail due to different code flow on `utils.opy::get_docker_client'Proposed solution
Adding three if statement to delete those environmental variables if they are not None, so the test function stay fresh in each run.