[DEV] switch nose with pytest #18025

szha · 2020-04-11T01:09:39Z

Description

switch nose with pytest

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented
To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

switch nose with pytest

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

mxnet-bot · 2020-04-11T01:09:43Z

Hey @szha , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

To trigger all jobs: @mxnet-bot run ci [all]
To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [centos-gpu, edge, windows-cpu, website, centos-cpu, unix-cpu, unix-gpu, windows-gpu, miscellaneous, sanity, clang]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

leezu · 2020-04-11T03:09:53Z

How about the random seed handling and reproducibility? I think you'll need to adapt https://github.com/dmlc/gluon-nlp/blob/v0.9.x/conftest.py#L41

szha · 2020-04-11T06:53:30Z

@leezu thanks. Added.

leezu · 2020-04-16T01:14:11Z

.github/workflows/os_x_staticbuild.yml

@@ -10,16 +10,16 @@ jobs:
        uses: actions/checkout@v2
      - name: Install Dependencies
        run: |
-          brew install nasm automake ninja libtool cmake pkgconfig protobuf
+          brew install nasm automake ninja libtool cmake pkgconfig protobuf hdf5 zlib
+          python3 -m pip install --user -r ci/docker/install/requirements


I suggest not to use ci/docker/install/requirements here but list the dependencies explicitly. Most of the things in ci/docker/install/requirements are not needed for this test and having a single requirements file that is used from an opaque number of locations for an variety of purposes makes it very hard to maintain that file.

Specifically, here we can just replace python3 -m pip install --user nose nose-timer nose-exclude numpy scipy with python3 -m pip install --user pytest numpy scipy a few lines below.

marcoabreu · 2020-04-22T18:08:54Z

@mxnet-bot run ci [unix-gpu]

mxnet-bot · 2020-04-22T18:09:01Z

Jenkins CI successfully triggered : [unix-gpu]

leezu · 2020-04-22T18:11:35Z

@szha there is a bug with the unix-gpu dist kvstore tests

Runtime on master is 6 minutes http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/master/1887/pipeline/426 but more than 1 hour in this PR http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18025/56/pipeline

leezu · 2020-04-22T18:12:34Z

Ok, the difference is due to hanging issue..

[2020-04-22T17:06:49.477Z] AssertionError: (None, array([[0.6, 0.6, 0.6],

[2020-04-22T17:06:49.477Z] [0.6, 0.6, 0.6]], dtype=float32), 41.0)

[2020-04-22T17:06:49.477Z] terminate called without an active exception

Thanks @marcoabreu to retrigger

marcoabreu · 2020-04-22T18:16:43Z

The stability of our test suite is really a pain in the butt :(

You're welcome

leezu · 2020-04-22T19:43:04Z

This does not seem to be a stability issue, but a bug. Same failure again:

[2020-04-22T19:21:39.589Z] Traceback (most recent call last):

[2020-04-22T19:21:39.589Z]   File "dist_device_sync_kvstore.py", line 130, in <module>

[2020-04-22T19:21:39.589Z]     test_sync_push_pull()

[2020-04-22T19:21:39.589Z]   File "dist_device_sync_kvstore.py", line 88, in test_sync_push_pull

[2020-04-22T19:21:39.589Z]     check_default_keys(kv, my_rank, nworker, nrepeat=3)

[2020-04-22T19:21:39.589Z]   File "dist_device_sync_kvstore.py", line 72, in check_default_keys

[2020-04-22T19:21:39.589Z]     check_diff_to_scalar(val, num)

[2020-04-22T19:21:39.589Z]   File "dist_device_sync_kvstore.py", line 29, in check_diff_to_scalar

[2020-04-22T19:21:39.589Z]     assert(np.sum(np.abs((A - x).asnumpy())) == 0), (rank, A.asnumpy(), x)

[2020-04-22T19:21:39.589Z] AssertionError: (None, array([[0.6, 0.6, 0.6],

[2020-04-22T19:21:39.589Z]        [0.6, 0.6, 0.6]], dtype=float32), 41.0)

@szha

leezu · 2020-04-22T21:49:15Z

Still

[2020-04-22T21:11:19.922Z] + cd /work/mxnet/tests/nightly

[2020-04-22T21:11:19.922Z] + ./test_distributed_training-gpu.sh

[2020-04-22T21:11:41.844Z] worker 0 is initialized

[2020-04-22T21:11:41.844Z] worker 0 passed test_gluon_trainer_type

[2020-04-22T21:11:41.844Z] worker 2 is initialized

[2020-04-22T21:11:41.844Z] worker 2 passed test_gluon_trainer_type

[2020-04-22T21:11:41.844Z] worker 1 is initialized

[2020-04-22T21:11:41.844Z] worker 1 passed test_gluon_trainer_type

[2020-04-22T21:11:41.844Z] worker 3 is initialized

[2020-04-22T21:11:41.844Z] worker 3 passed test_gluon_trainer_type

[2020-04-22T21:11:41.844Z] terminate called without an active exception

[2020-04-22T21:11:41.844Z] terminate called without an active exception

[2020-04-22T21:11:41.844Z] terminate called without an active exception

[2020-04-22T21:11:41.844Z] terminate called without an active exception

ci/jenkins/Jenkinsfile_centos_cpu

yzhliu

good to me

marcoabreu · 2020-04-23T08:35:54Z

Woohoo!

ChaiBapchya · 2020-04-30T20:32:01Z

ci/docker/runtime_functions.sh


    # https://github.com/apache/incubator-mxnet/issues/11801
    # if [[ ${mxnet_variant} = "cpu" ]] || [[ ${mxnet_variant} = "mkl" ]]; then
        # integrationtest_ubuntu_cpu_dist_kvstore
    # fi

    if [[ ${mxnet_variant} = cu* ]]; then
-        $nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/gpu
+        pytest --durations=50 --verbose tests/python/gpu


why are we showing top 50 slowest duration only?

Why not? The others are not really of interest

* switch nose with pytest * switch centos python to 3.6 * disable dist kvstore tests * skip hanging test

szha requested review from aaronmarkham, eric-haibin-lin, gigasquid and marcoabreu as code owners April 11, 2020 01:09

szha force-pushed the pytest branch 3 times, most recently from 2c73dcf to ccf2342 Compare April 11, 2020 02:58

szha force-pushed the pytest branch 3 times, most recently from d09779c to b7ab4a2 Compare April 11, 2020 06:53

szha force-pushed the pytest branch from b7ab4a2 to dc262eb Compare April 11, 2020 20:22

szha requested a review from leezu as a code owner April 11, 2020 20:22

szha force-pushed the pytest branch 14 times, most recently from f94b92f to e848924 Compare April 15, 2020 23:05

leezu reviewed Apr 16, 2020

View reviewed changes

szha added 2 commits April 22, 2020 12:44

switch nose with pytest

fa0b63f

switch centos python to 3.6

6e6584f

szha force-pushed the pytest branch from 645d508 to 7f83c3f Compare April 22, 2020 19:59

disable dist kvstore tests

8bdb6df

szha force-pushed the pytest branch from 7f83c3f to 8bdb6df Compare April 22, 2020 21:59

leezu reviewed Apr 23, 2020

View reviewed changes

ci/jenkins/Jenkinsfile_centos_cpu Show resolved Hide resolved

skip hanging test

5c10b89

yzhliu approved these changes Apr 23, 2020

View reviewed changes

szha merged commit faccd91 into apache:master Apr 23, 2020

szha deleted the pytest branch April 23, 2020 06:59

This was referenced Apr 23, 2020

caffe test activation or removal #18150

Closed

Disable tests of numpy operators implemented with CustomOp #18151

Merged

Fix CI reliance on executable bits #18159

Merged

haojin2 mentioned this pull request Apr 30, 2020

with_seed() broken when running GPU unit test #18198

Open

ChaiBapchya reviewed Apr 30, 2020

View reviewed changes

ruro mentioned this pull request May 27, 2020

Fix tests for ONNX version 1.5.0 bump #18054

Merged

MoisesHer mentioned this pull request Jun 23, 2020

Cherry-pick #18310 #18355 #18608

Merged

AntiZpvoh pushed a commit to AntiZpvoh/incubator-mxnet that referenced this pull request Jul 6, 2020

[DEV] switch nose with pytest (apache#18025)

adbdc47

* switch nose with pytest * switch centos python to 3.6 * disable dist kvstore tests * skip hanging test

leezu mentioned this pull request Jul 25, 2020

Remove caffe plugin #18787

Merged

leezu mentioned this pull request Oct 12, 2020

Remove duplicate @with_seed decorators #19336

Merged

waytrue17 mentioned this pull request Mar 23, 2021

[v1.x] Rearrange ONNX tests in Nightly CI #20075

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DEV] switch nose with pytest #18025

[DEV] switch nose with pytest #18025

szha commented Apr 11, 2020

mxnet-bot commented Apr 11, 2020

leezu commented Apr 11, 2020

szha commented Apr 11, 2020

leezu Apr 16, 2020 •

edited

Loading

marcoabreu commented Apr 22, 2020

mxnet-bot commented Apr 22, 2020

leezu commented Apr 22, 2020

leezu commented Apr 22, 2020

marcoabreu commented Apr 22, 2020

leezu commented Apr 22, 2020

leezu commented Apr 22, 2020

yzhliu left a comment

marcoabreu commented Apr 23, 2020

ChaiBapchya Apr 30, 2020

marcoabreu May 1, 2020

[DEV] switch nose with pytest #18025

[DEV] switch nose with pytest #18025

Conversation

szha commented Apr 11, 2020

Description

Checklist

Essentials

Changes

Comments

mxnet-bot commented Apr 11, 2020

leezu commented Apr 11, 2020

szha commented Apr 11, 2020

leezu Apr 16, 2020 • edited Loading

Choose a reason for hiding this comment

marcoabreu commented Apr 22, 2020

mxnet-bot commented Apr 22, 2020

leezu commented Apr 22, 2020

leezu commented Apr 22, 2020

marcoabreu commented Apr 22, 2020

leezu commented Apr 22, 2020

leezu commented Apr 22, 2020

yzhliu left a comment

Choose a reason for hiding this comment

marcoabreu commented Apr 23, 2020

ChaiBapchya Apr 30, 2020

Choose a reason for hiding this comment

marcoabreu May 1, 2020

Choose a reason for hiding this comment

leezu Apr 16, 2020 •

edited

Loading