Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

CUDA: unspecified launch failure on CI Windows #17616

Closed
szhengac opened this issue Feb 17, 2020 · 5 comments
Closed

CUDA: unspecified launch failure on CI Windows #17616

szhengac opened this issue Feb 17, 2020 · 5 comments
Labels
CI Flaky

Comments

@szhengac
Copy link
Contributor

Description

CI on windows keeps giving CUDA: unspecified launch failure. It has been retriggered many times.

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17400/20/pipeline/
@szhengac szhengac added the Flaky label Feb 17, 2020
@ptrendx
Copy link
Member

ptrendx commented Feb 19, 2020

@haojin2 The first failing test is test_operator_gpu.test_np_nan_to_num.

@haojin2
Copy link
Contributor

haojin2 commented Feb 19, 2020 •
edited
Loading

Please first disable then I’ll investigate.

@haojin2
Copy link
Contributor

haojin2 commented Feb 20, 2020

#17630 still fails on windows-gpu with test_np_nan_to_num disabled, the error starts to occur on the test immediately after test_np_nan_to_num, meanwhile there were other PRs passing the windows-gpu build, as a result, I suspect the problem is not with the test_np_nan_to_num itself.

@ChaiBapchya
Copy link
Contributor

Another one for unrelated PR #17487 - http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17487/24/pipeline

@szha szha added the CI label Apr 22, 2020
@leezu
Copy link
Contributor

leezu commented Apr 24, 2020

This issue appears fixed by the toolchain updgrade #17962

@leezu leezu closed this as completed Apr 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
CI Flaky
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@leezu @szha @szhengac @ptrendx @ChaiBapchya @haojin2