[numpy] Fix unit tests after introducing numpy compatible shapes #14487

reminisce · 2019-03-21T07:18:07Z

Fixed a portion of failing tests after introducing numpy compatible shapes.
Added a thread-safe switch to turn on/off numpy compatibility. By default, it's off and existing tests should not be affected.

@junrushao1994 @szha @eric-haibin-lin @zheng-da @yzhliu

python/mxnet/rnn/rnn_cell.py

zheng-da · 2019-03-21T20:41:22Z

src/c_api/c_api_common.h

@@ -91,7 +91,7 @@ struct MXAPIThreadLocalEntry {
    data->resize(shapes.size());
    size_t size = 0;
    for (const auto& s : shapes) {
-      CHECK_GE(s.ndim(), 0);
+      if (s.ndim() > 0);


what is this?

s.ndim() can be -1 if the shape is unknown. In that situation, it should not contribute to the calculation of the total size.

why not remove this line of code?

src/operator/batch_norm_v1-inl.h

zheng-da · 2019-03-21T20:51:33Z

src/operator/contrib/multi_proposal-inl.h

@@ -108,7 +108,7 @@ class MultiProposalProp : public OperatorProperty {
    using namespace mshadow;
    CHECK_EQ(in_shape->size(), 3) << "Input:[cls_prob, bbox_pred, im_info]";
    const mxnet::TShape &dshape = in_shape->at(proposal::kClsProb);
-    if (!mxnet::op::shape_is_none(dshape)) return false;
+    if (mxnet::op::shape_is_none(dshape)) return false;


this simply reverses the condition check?

Yes, it was done incorrectly in a previous PR.

szha

Otherwise, looks good to me

python/mxnet/rnn/rnn_cell.py

junrushao

LGTM

) * Fix infer shape rnn * Fix boolean mask and custom op unit tests * Fix multi proposal * Fix diag * Add global switch for backward compatibility and fix infer shape bugs * Fix slice op infer shape * Fix rnn infer shape * Add util funcs for ndim_is_known and dim_size_is_known * Revert rnn_cell.py

…che#14487) * Fix infer shape rnn * Fix boolean mask and custom op unit tests * Fix multi proposal * Fix diag * Add global switch for backward compatibility and fix infer shape bugs * Fix slice op infer shape * Fix rnn infer shape * Add util funcs for ndim_is_known and dim_size_is_known * Revert rnn_cell.py

) * Fix infer shape rnn * Fix boolean mask and custom op unit tests * Fix multi proposal * Fix diag * Add global switch for backward compatibility and fix infer shape bugs * Fix slice op infer shape * Fix rnn infer shape * Add util funcs for ndim_is_known and dim_size_is_known * Revert rnn_cell.py

…che#14487) * Fix infer shape rnn * Fix boolean mask and custom op unit tests * Fix multi proposal * Fix diag * Add global switch for backward compatibility and fix infer shape bugs * Fix slice op infer shape * Fix rnn infer shape * Add util funcs for ndim_is_known and dim_size_is_known * Revert rnn_cell.py

* [numpy] Shape support scalar tensor (#14315) * Support scalar and zero-size tensors with np.sum * Add sanity check when ndim is set * [Numpy] Change semantics of ndim for operators in `src/operator/contrib` (#14409) * Initial commit * Address comments * [WIP] Use new shape definition (#14453) * Init checkin * Fix ndarray alloc bug * Use TShape(0) as default empty tuple params * Fix bugs * Fix TShape init value * Fix infer shape pass shape type and reshape infer shape func * [numpy] Fix unit tests after introducing numpy compatible shapes (#14487) * Fix infer shape rnn * Fix boolean mask and custom op unit tests * Fix multi proposal * Fix diag * Add global switch for backward compatibility and fix infer shape bugs * Fix slice op infer shape * Fix rnn infer shape * Add util funcs for ndim_is_known and dim_size_is_known * Revert rnn_cell.py * Fix a bug to pass the test in test_contrib_rnn (#14520) * fix. * remove type conversion. * remove type cast. * [numpy] Fix test_dynamic_shape.test_dynamic_shape (#14538) * Initial commit * Address comments from Jun * [numpy] Fix numpy import in python2 (#14537) * Fix several test failures * Fix subgraph op infer shape * Fix sparse slice * Fix deconv infer shape * Fix numpy import compatibility problem in python2 * fix concat and slice (#14549) * fix R-package (#14536) * Fix cpp package build after using new shape definition (#14554) * Fix pooling_v1 and deformable_convolution param initialization (#14577) * Fix pooling_v1 param initialization * Fix deformable_convolution param initialization * [Numpy] Misc fix (#14612) * [Numpy] Misc Fix * fix build * !shape_is_none => shape_is_known * Address comments * Fix * [Numpy] fix test_operator_gpu.test_upsampling_bilinear_with_type (#14557) * Fix test_operator_gpu.test_upsampling_bilinear_with_type * Address comments * [Numpy] Java/Scala modification (#14625) * modify jni to support 0 dim/shape * fix transpose axes default value * fix shape index bug (#14630) * fix jni lint (#14634) * [numpy] Fix numpy branch failing tests in CI (#14639) * Remove numpy namespaces for operator registration * Fix bug when shape is compeltely unknown * Fix singed/unsigned compare warning * Fix CI * Fix pylint * Avoid launching gpu kernels for zero-size output tensors * Fix test_ndarray * Fix binary broadcast with zero-size tensors * Better error message for infer shape failure in imperative * Fix TShape constructor ambiguity on certain platforms * Fix mkldnn build failure * Fix build failure in gpu and cpp test * Fix gpu cpp test build with mkldnn * Fix mkldnn cpp test * Fix concatenating zero-size tensors * Avoid letting mkldnn handle zero-size tensors in concat * Fix quantized_concat infer shape * Try to fix perl c api * fix invalid ndarray dispose (#14657) * swig fixes for the changes in c_api.h (#14655) * Rename np_comp to np_compat for readability * Fix import error * Keep old c apis unchanged * Fix lint * Rebase and fix build * Fix R build failure * Fix Perl build failure * Rebase with master * Address cr comments * Use just one scope to represent numpy compatibility * Add code comment to NumpyScope object in Scala * Add use_np_compat decorator * Fix pylint

* [numpy] Shape support scalar tensor (apache#14315) * Support scalar and zero-size tensors with np.sum * Add sanity check when ndim is set * [Numpy] Change semantics of ndim for operators in `src/operator/contrib` (apache#14409) * Initial commit * Address comments * [WIP] Use new shape definition (apache#14453) * Init checkin * Fix ndarray alloc bug * Use TShape(0) as default empty tuple params * Fix bugs * Fix TShape init value * Fix infer shape pass shape type and reshape infer shape func * [numpy] Fix unit tests after introducing numpy compatible shapes (apache#14487) * Fix infer shape rnn * Fix boolean mask and custom op unit tests * Fix multi proposal * Fix diag * Add global switch for backward compatibility and fix infer shape bugs * Fix slice op infer shape * Fix rnn infer shape * Add util funcs for ndim_is_known and dim_size_is_known * Revert rnn_cell.py * Fix a bug to pass the test in test_contrib_rnn (apache#14520) * fix. * remove type conversion. * remove type cast. * [numpy] Fix test_dynamic_shape.test_dynamic_shape (apache#14538) * Initial commit * Address comments from Jun * [numpy] Fix numpy import in python2 (apache#14537) * Fix several test failures * Fix subgraph op infer shape * Fix sparse slice * Fix deconv infer shape * Fix numpy import compatibility problem in python2 * fix concat and slice (apache#14549) * fix R-package (apache#14536) * Fix cpp package build after using new shape definition (apache#14554) * Fix pooling_v1 and deformable_convolution param initialization (apache#14577) * Fix pooling_v1 param initialization * Fix deformable_convolution param initialization * [Numpy] Misc fix (apache#14612) * [Numpy] Misc Fix * fix build * !shape_is_none => shape_is_known * Address comments * Fix * [Numpy] fix test_operator_gpu.test_upsampling_bilinear_with_type (apache#14557) * Fix test_operator_gpu.test_upsampling_bilinear_with_type * Address comments * [Numpy] Java/Scala modification (apache#14625) * modify jni to support 0 dim/shape * fix transpose axes default value * fix shape index bug (apache#14630) * fix jni lint (apache#14634) * [numpy] Fix numpy branch failing tests in CI (apache#14639) * Remove numpy namespaces for operator registration * Fix bug when shape is compeltely unknown * Fix singed/unsigned compare warning * Fix CI * Fix pylint * Avoid launching gpu kernels for zero-size output tensors * Fix test_ndarray * Fix binary broadcast with zero-size tensors * Better error message for infer shape failure in imperative * Fix TShape constructor ambiguity on certain platforms * Fix mkldnn build failure * Fix build failure in gpu and cpp test * Fix gpu cpp test build with mkldnn * Fix mkldnn cpp test * Fix concatenating zero-size tensors * Avoid letting mkldnn handle zero-size tensors in concat * Fix quantized_concat infer shape * Try to fix perl c api * fix invalid ndarray dispose (apache#14657) * swig fixes for the changes in c_api.h (apache#14655) * Rename np_comp to np_compat for readability * Fix import error * Keep old c apis unchanged * Fix lint * Rebase and fix build * Fix R build failure * Fix Perl build failure * Rebase with master * Address cr comments * Use just one scope to represent numpy compatibility * Add code comment to NumpyScope object in Scala * Add use_np_compat decorator * Fix pylint

reminisce added 6 commits March 18, 2019 23:24

Fix infer shape rnn

0f44298

Fix boolean mask and custom op unit tests

70434d6

Fix multi proposal

5d37715

Fix diag

676495f

Add global switch for backward compatibility and fix infer shape bugs

7ae5eb7

Fix slice op infer shape

b5a683d

reminisce added the Numpy label Mar 21, 2019

reminisce requested review from anirudh2290 and szha as code owners March 21, 2019 07:18

reminisce commented Mar 21, 2019

View reviewed changes

python/mxnet/rnn/rnn_cell.py Outdated Show resolved Hide resolved

Fix rnn infer shape

ca6e37b

reminisce changed the title ~~[WIP][numpy] Fix unit tests after introducing numpy compatible shapes~~ [numpy] Fix unit tests after introducing numpy compatible shapes Mar 21, 2019

zheng-da reviewed Mar 21, 2019

View reviewed changes

Add util funcs for ndim_is_known and dim_size_is_known

949f35a

szha approved these changes Mar 22, 2019

View reviewed changes

python/mxnet/rnn/rnn_cell.py Outdated Show resolved Hide resolved

junrushao approved these changes Mar 22, 2019

View reviewed changes

Revert rnn_cell.py

7e0d9ce

szha merged commit 64c61b9 into apache:numpy Mar 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[numpy] Fix unit tests after introducing numpy compatible shapes #14487

[numpy] Fix unit tests after introducing numpy compatible shapes #14487

reminisce commented Mar 21, 2019

zheng-da Mar 21, 2019

reminisce Mar 21, 2019

eric-haibin-lin Mar 24, 2019

zheng-da Mar 21, 2019

reminisce Mar 21, 2019

szha left a comment

junrushao left a comment

[numpy] Fix unit tests after introducing numpy compatible shapes #14487

[numpy] Fix unit tests after introducing numpy compatible shapes #14487

Conversation

reminisce commented Mar 21, 2019

zheng-da Mar 21, 2019

Choose a reason for hiding this comment

reminisce Mar 21, 2019

Choose a reason for hiding this comment

eric-haibin-lin Mar 24, 2019

Choose a reason for hiding this comment

zheng-da Mar 21, 2019

Choose a reason for hiding this comment

reminisce Mar 21, 2019

Choose a reason for hiding this comment

szha left a comment

Choose a reason for hiding this comment

junrushao left a comment

Choose a reason for hiding this comment