Fix large array tests #16328

ChaiBapchya · 2019-09-29T06:11:12Z

Description

fix activation, copy_to, random_multinomial
removed 2 redundant functions (imod, expand_dims)
Lint fixes
Random_* functions used

MEDIUM_X, SMALL_Y, SMALL_X, SMALL_Y -> (10000, 50, 100, 50) < 2**32 elements

Hence made it

MEDIUM_X, SMALL_X, SMALL_X, SMALL_Y -> (10000, 100, 100, 50) > 2**32 elements

pooling, on other hand, used

MEDIUM_X, MEDIUM_X, SMALL_Y, SMALL_Y -> (10000, 10000, 50, 50) >> 2**32 elements

A very big value that is > CPU memory gives the error

mxnet.base.MXNetError: [20:08:27] ../src/storage/./cpu_device_storage.h:75: Failed to allocate CPU Memory

Hence made it just greater than 2**32

MEDIUM_X, MEDIUM_X, SMALL_Y, SMALL_Y -> (10000, 200, 50, 50) > 2**32 elements

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

tests/nightly/test_large_array.py

Comments

marcoabreu · 2019-09-30T14:00:23Z

Please remove commented out code

tests/nightly/test_large_array.py

access2rohit · 2019-10-04T16:39:23Z

Thanks for the quick fix! I left 1 comment. Rest LGTM! Post complete test run for test_large_array.py here, then it should be good to go.

…ions

ChaiBapchya · 2019-10-09T21:58:02Z

test_large_array.test_gluon_embedding ... ok
test_large_array.test_ndarray_zeros ... ok
test_large_array.test_ndarray_ones ... ok
test_large_array.test_ndarray_convert ... ok
test_large_array.test_ndarray_random_uniform ... ok
test_large_array.test_ndarray_random_randint ... ok
test_large_array.test_ndarray_random_exponential ... ok
test_large_array.test_ndarray_random_gamma ... ok
test_large_array.test_ndarray_random_multinomial ... ok
test_large_array.test_ndarray_random_generalized_negative_binomial ... ok
test_large_array.test_ndarray_random_negative_binomial ... ok
test_large_array.test_ndarray_random_normal ... ok
test_large_array.test_ndarray_random_poisson ... ok
test_large_array.test_ndarray_random_randn ... ok
test_large_array.test_ndarray_random_shuffle ... ok
test_large_array.test_ndarray_empty ... ok
test_large_array.test_elementwise ... ok
test_large_array.test_reduce ... ok
test_large_array.test_dot ... ok
test_large_array.test_FullyConnected ... ok
test_large_array.test_broadcast ... ok
test_large_array.test_clip ... ok
test_large_array.test_split ... ok
test_large_array.test_argmin ... ok
test_large_array.test_tile ... ok
test_large_array.test_take ... ok
test_large_array.test_slice ... ok
test_large_array.test_slice_assign ... ok
test_large_array.test_expand_dims ... ok
test_large_array.test_squeeze ... ok
test_large_array.test_broadcast_div ... ok
test_large_array.test_Dense ... ok
test_large_array.test_where ... ok
test_large_array.test_pick ... ok
test_large_array.test_depthtospace ...Killed

Gets killed as it reaches max memory limit 480G/480G of p3 instance
Resume running from depthtospace

test_large_array.test_depthtospace ...
ok
test_large_array.test_spacetodepth ... ok
test_large_array.test_diag ... ok
test_large_array.test_ravel_multi_index ... ok
test_large_array.test_unravel_index ... ok
test_large_array.test_transpose ... ok
test_large_array.test_swapaxes ... ok
test_large_array.test_flip ... ok
test_large_array.test_softmax ... ok
test_large_array.test_argsort ... ok
test_large_array.test_sort ... ok
test_large_array.test_topk ... ok
test_large_array.test_exponent_logarithm_operators ... ok
test_large_array.test_power_operators ... ok
test_large_array.test_sequence_mask ... ERROR
test_large_array.test_sequence_reverse ... ok
test_large_array.test_sequence_last ... ok
test_large_array.test_softmax_cross_entropy ... ERROR
test_large_array.test_index_copy ... ok
test_large_array.testSoftmaxOutput ... [23:22:11] ../src/executor/graph_executor.cc:2014: Subgraph backend MKLDNN is activated.
ok
test_large_array.test_leaky_relu ... ok
test_large_array.test_pooling ... ok
test_large_array.test_layer_norm ... [23:34:54] ../src/executor/graph_executor.cc:1936: Subgraph backend MKLDNN is activated.
ERROR
test_large_array.test_dropout ... [23:35:57] ../src/executor/graph_executor.cc:1936: Subgraph backend MKLDNN is activated.
ok
test_large_array.test_activation ... ok
test_large_array.test_batchnorm ... ok
test_large_array.test_add ... ok
test_large_array.test_sub ... ok
test_large_array.test_rsub ... ok
test_large_array.test_neg ... ok
test_large_array.test_mul ... ok
test_large_array.test_div ... ok
test_large_array.test_rdiv ... ok
test_large_array.test_mod ... ok
test_large_array.test_rmod ... ok
test_large_array.test_imod ... ok
test_large_array.test_pow ... ok
test_large_array.test_rpow ... ok
test_large_array.test_shape ... ERROR
test_large_array.test_size ... ok
test_large_array.test_copy ... ok
test_large_array.test_copy_to ... ok
test_large_array.test_zeros_like ... Killed

access2rohit · 2019-10-10T17:58:10Z

@anirudh2290 can you review and merge this

anirudh2290 · 2019-10-10T18:02:07Z

There are a few tests failing after you resumed running. Why ?

test_large_array.test_softmax_cross_entropy ... ERROR

ChaiBapchya · 2019-10-10T18:16:33Z

Tests that give error when run together, pass when run individually.

nosetests tests/nightly/test_large_array.py:test_softmax_cross_entropy --verbose -s
test_large_array.test_softmax_cross_entropy ... ok

----------------------------------------------------------------------
Ran 1 test in 199.836s

OK

I'm guessing memory overflow has to do with this (not sure though) But we've seen this since quite sometime

tests getting killed
tests error out when run together but individually they pass

marcoabreu · 2019-10-10T20:56:44Z

I assume you're testing with the teardown for now. But before making the PR final, I'd appreciate if you could elaborate. Feel free to ping me once you figured it out.

access2rohit · 2019-10-11T20:25:38Z

I assume you're testing with the teardown for now. But before making the PR final, I'd appreciate if you could elaborate. Feel free to ping me once you figured it out.

@marcoabreu : we need teardown to free up memory allocated once each test finishes. The reason of doing this is to ensure that memory is freed up after each test run, which wasn't happening before. These tests require more memory, running all of them in a single execution was causing out of memory error since memory was not being freed up in time(as observed using htop). Therefore we need to call clear cache and teardown after individual test runs. Let me know if it makes sense.

anirudh2290 · 2019-10-11T20:56:09Z

Is there a issue for the disabled nightly large tensor tests ? If not can you please open an issue for these disabled tests.

ChaiBapchya · 2019-10-11T21:22:11Z

~~Nope. I made one to document these findings. #16447~~
This one - #14980

anirudh2290 · 2019-10-11T21:28:40Z

also why does the title still say WIP

anirudh2290 · 2019-10-11T21:30:19Z

tests/python/unittest/common.py

+                raise
+            finally:
+                mx.nd.waitall()
+                mx.cpu().empty_cache()


are the tests only for cpu context ?

Ya. We don't test for gpu (as large tensor GPU is not supported)

* fix activation * remove comments * fix copy_to * fix lint, remove redundant function, fix shape sizes for random functions * fix sigmoid issue * fix leaky relu * fix random shuffle * fix pooling * fix dropout * fix index copy * add teardown and fix lint * post test cleanup * removed decorator since it needs C API for CPU memory release

This was referenced Oct 2, 2019

test_operator_gpu.test_kernel_error_checking Fails #16353

Open

Win10 build error: gtest, no 'object' file generated #14378

Open

access2rohit reviewed Oct 4, 2019

View reviewed changes

tests/nightly/test_large_array.py Show resolved Hide resolved

ChaiBapchya force-pushed the fix_large_array branch 2 times, most recently from 8d3ea39 to fd69dfb Compare October 4, 2019 18:18

ChaiBapchya added 10 commits October 9, 2019 13:33

fix activation

1684abe

remove comments

e54f410

fix copy_to

575bf5a

fix lint, remove redundant function, fix shape sizes for random funct…

cc5daf8

…ions

fix sigmoid issue

db628d7

fix leaky relu

2c8a08e

fix random shuffle

b57b884

fix pooling

b448a70

fix dropout

087f20a

fix index copy

3a4025f

ChaiBapchya force-pushed the fix_large_array branch from 1399bbd to 3a4025f Compare October 9, 2019 20:34

ChaiBapchya mentioned this pull request Oct 9, 2019

[Flaky] test_np_max_min on windows #16389

Closed

access2rohit approved these changes Oct 10, 2019

View reviewed changes

add teardown and fix lint

50b40f9

marcoabreu changed the title ~~Fix large array tests~~ [WIP] Fix large array tests Oct 10, 2019

marcoabreu added the pr-work-in-progress PR is still work in progress label Oct 10, 2019

post test cleanup

c48f70f

ChaiBapchya changed the title ~~[WIP] Fix large array tests~~ Fix large array tests Oct 11, 2019

anirudh2290 reviewed Oct 11, 2019

View reviewed changes

removed decorator since it needs C API for CPU memory release

0361959

anirudh2290 approved these changes Oct 14, 2019

View reviewed changes

anirudh2290 merged commit 858a52e into apache:master Oct 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix large array tests #16328

Fix large array tests #16328

ChaiBapchya commented Sep 29, 2019 •

edited

Loading

marcoabreu commented Sep 30, 2019

access2rohit commented Oct 4, 2019

ChaiBapchya commented Oct 9, 2019 •

edited

Loading

access2rohit commented Oct 10, 2019

anirudh2290 commented Oct 10, 2019

ChaiBapchya commented Oct 10, 2019 •

edited

Loading

marcoabreu commented Oct 10, 2019

access2rohit commented Oct 11, 2019

anirudh2290 commented Oct 11, 2019 •

edited

Loading

ChaiBapchya commented Oct 11, 2019 •

edited

Loading

anirudh2290 commented Oct 11, 2019

anirudh2290 Oct 11, 2019

ChaiBapchya Oct 11, 2019 •

edited

Loading

Fix large array tests #16328

Fix large array tests #16328

Conversation

ChaiBapchya commented Sep 29, 2019 • edited Loading

Description

Checklist

Essentials

Changes

Comments

marcoabreu commented Sep 30, 2019

access2rohit commented Oct 4, 2019

ChaiBapchya commented Oct 9, 2019 • edited Loading

access2rohit commented Oct 10, 2019

anirudh2290 commented Oct 10, 2019

ChaiBapchya commented Oct 10, 2019 • edited Loading

marcoabreu commented Oct 10, 2019

access2rohit commented Oct 11, 2019

anirudh2290 commented Oct 11, 2019 • edited Loading

ChaiBapchya commented Oct 11, 2019 • edited Loading

anirudh2290 commented Oct 11, 2019

anirudh2290 Oct 11, 2019

Choose a reason for hiding this comment

ChaiBapchya Oct 11, 2019 • edited Loading

Choose a reason for hiding this comment

ChaiBapchya commented Sep 29, 2019 •

edited

Loading

ChaiBapchya commented Oct 9, 2019 •

edited

Loading

ChaiBapchya commented Oct 10, 2019 •

edited

Loading

anirudh2290 commented Oct 11, 2019 •

edited

Loading

ChaiBapchya commented Oct 11, 2019 •

edited

Loading

ChaiBapchya Oct 11, 2019 •

edited

Loading