Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[CI] Flaky test in CI windows-gpu #20337

Open
Kh4L opened this issue Jun 8, 2021 · 2 comments
Open

[CI] Flaky test in CI windows-gpu #20337

Kh4L opened this issue Jun 8, 2021 · 2 comments
Labels

Comments

@Kh4L
Copy link
Contributor

Kh4L commented Jun 8, 2021

Description

windows-gpu CI has some flaky tests.

03:18:11  FAILED tests/python/gpu/test_operator_gpu.py::test_sparse_nd_pickle - mxnet.b...
03:18:11  FAILED tests/python/gpu/test_operator_gpu.py::test_sparse_nd_save_load[save]
03:18:11  FAILED tests/python/gpu/test_operator_gpu.py::test_sparse_ndarray_load_csr_npz_scipy[save]

Occurrences

https://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/job/windows-gpu/job/master/2524/execution/node/155/log/

@Kh4L Kh4L added the Flaky label Jun 8, 2021
@TristonC
Copy link
Contributor

@leezu Could you find someone to help?

@DickJC123
Copy link
Contributor

I ran into failures on these exact 3 tests. I've tracked down the problem: the unittest test_zero_sized_dim does not clean up after itself, and leaves is_np_shape_thread_local_ = 1. Subsequent tests run by the same pytest worker might fail as a result. So I get a failure on:

pytest --verbose -s tests/python/gpu/test_operator_gpu.py::{test_zero_sized_dim,test_sparse_nd_pickle}

but not on:

pytest --verbose -s tests/python/gpu/test_operator_gpu.py::test_sparse_nd_pickle

Looking into the most appropriate fix...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants