Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Boolean indexing accesses out of bound elements #17990

Closed
leezu opened this issue Apr 7, 2020 · 3 comments
Closed

Boolean indexing accesses out of bound elements #17990

leezu opened this issue Apr 7, 2020 · 3 comments
Labels
Bug Disabled test

Comments

@leezu
Copy link
Contributor

leezu commented Apr 7, 2020 •
edited
Loading

Description

CI with updated toolchain (ie #17984) catches the bug.

vector: :_M_range_check: __n (which is 2) >= this->size() (which is 2)

Error Message

To Reproduce

Build with this simple patch

diff --git a/src/operator/numpy/np_boolean_mask_assign.cc b/src/operator/numpy/np_boolean_mask_assign.cc
index e01ebb7c6..d5ab00835 100644
--- a/src/operator/numpy/np_boolean_mask_assign.cc
+++ b/src/operator/numpy/np_boolean_mask_assign.cc
@@ -220,7 +220,7 @@ void NumpyBooleanAssignForwardCPU(const nnvm::NodeAttrs& attrs,
   // If there's no True in mask, return directly
   if (valid_num == 0) return;

-  const TShape& vshape = inputs[2].shape_;
+  const TShape& vshape = inputs.at(2).shape_;

   if (inputs.size() == 3U) {
     // tensor case

OR follow the instructions in #17987 to trigger this via glibc assertions in debug build.

Run test_np_ndarray_boolean_indexing, test_npx_batch_dot or test_npi_boolean_assign to trigger the bug.

CC: @haojin2
@leezu leezu added the Bug label Apr 7, 2020
@leezu leezu mentioned this issue Apr 12, 2020
Merged
leezu added a commit that referenced this issue Apr 14, 2020
@leezu
Switch to C++17 and modernize toolchain + CI (#17984)
fb73a17 
As per #17968, require C++17 compatible compiler. For cuda code, use C++14 mode introduced in Cuda 9. C++17 support for Cuda will be available in Cuda 11.

Switching to C++17 requires modernizing the toolchain, which exposed a number  of technical debt issues in the codebase. All blocking issues are fixed as part of this PR. See the full list below.

This PR contains the following specific changes:

    Switch CI pipeline to use gcc7 on Ubuntu and CentOS
    Switch CD pipeline to CentOS 7 with https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/ This enables us to build with gcc7 C++17 compiler while keeping a relatively old glibc requirement for distribution.
    Simplify ARM Edge builds
        Switch to standard Ubuntu / Debian cross-compilation toolchain for ARMv7, ARMv8
        Switch to https://toolchains.bootlin.com/ toolchain for ARMv6 (the Debian ARMv6 toolchain is for ARMv4 + ARMv5 + ARMv6, but we wish to only target ARMv6 and make use of ARMv6 features)
        Remove reliance on dockcross for cross compilation.
    Simplify Jetson build
        Use standard Ubuntu / Debian cross-compilation toolchain for ARMv8
        Upgrade to Cuda 10 and Jetpack 4.3
        Simplify build setup
    Simplify QEMU ARM virtualization test setup on CI
        Remove complex "Virtual Machine in Docker" logic and run a QEMU based Docker container instead based on arm32v7/ubuntu
    Fix out of bounds vector accesses in
        SoftmaxGradOpType
        MKLDNNFCBackward
    Fix use of non-standard rand_r function (which is not available on anymore on newer Android toolchains and shouldn't be use in any case).
    Fix reproducibility of RNN with Dropout
    Fix reproducibility of DGL Graph Sampling Operators
    Update tests for Android Edge build to NDK19. The previously used standalone toolchain is obsolete.

Those Dockerfiles that required refactoring as part of the effort were refactored based on the following consideration

    Maximize the use of system dependencies provided by the distribution instead of manually installing dependencies from source or from third party vendors. This reduces the complexity of the installation process and essentially pins the dependency versions, increasing CI stability. Further, Dockerfile build speed is improved. To facilitate this, use recent distribution versions. We still ensure backwards compatibility via CentOS7 based build and test stages
    Minimize the number of layers in the Dockerfile. Don't have 5 different script files executed, each calling apt-get update and install, but just execute once. Speeds up the build and reduces image size. Keep each Dockerfile simple and tailored to a purpose, instead of running 20 scripts to install dependencies for every thinkable scenario, which is unmaintainable.

Some more small changes:

    Remove outdated references to Cuda 7 and Cuda 8 in various files.
    Remove C++03 support in mshadow
    Disable broken tests
        NumpyBooleanAssignForwardCPU #17990
        test_init.test_rsp_const_init #17988
        quantized_elemwise_mul #18034

List of squashed commits

* cpp standard

* Remove leftover files of Cuda 7 and Cuda 8 support

* thrust 1.9.8 for clang10

* compiler warnings

* Disable broken test_init.test_rsp_const_init

* Disable tests invoking NumpyBooleanAssignForwardCPU

* Fix out of bounds access in SoftmaxGradOpType

* Use CentOS 7 for staticbuilds

CentOS 7 fullfills the requirements for PEP 599 manylinux-2014 and provides a
C++17 toolchain.

* Fix MKLDNNFCBackward

* Update edge toolchain

* Support platforms without rand_r

* Cleanup random.h

* Greatly simplify qemu setup

* Remove unused functions in Jenkins_steps.groovy

* Skip quantized_elemwise_mul due QuantizedElemwiseMulOpShape bug

* Fix R package installation

#18042

* Fix centos ccache

* Fix GPU Makefile staticbuild on CentOS7

* CentOS7 NCCL

* CentOS7 staticbuild fix link with libculibos
@leezu
Copy link
Contributor Author

leezu commented Apr 15, 2020

@haojin2 will you fix the bug and reenable the tests?

@haojin2
Copy link
Contributor

haojin2 commented Apr 17, 2020

Fixed in #17796, test re-enabled

@Yiyan66 Yiyan66 mentioned this issue Apr 24, 2020
Merged
7 tasks
@yzhliu
Copy link
Member

yzhliu commented Apr 27, 2020 •
edited
Loading

should be fixed by #18158 (originally #17796)

@yzhliu yzhliu closed this as completed Apr 27, 2020
AntiZpvoh pushed a commit to AntiZpvoh/incubator-mxnet that referenced this issue Jul 6, 2020
@leezu @AntiZpvoh
Switch to C++17 and modernize toolchain + CI (apache#17984)
d90f0e5 
As per apache#17968, require C++17 compatible compiler. For cuda code, use C++14 mode introduced in Cuda 9. C++17 support for Cuda will be available in Cuda 11.

Switching to C++17 requires modernizing the toolchain, which exposed a number  of technical debt issues in the codebase. All blocking issues are fixed as part of this PR. See the full list below.

This PR contains the following specific changes:

    Switch CI pipeline to use gcc7 on Ubuntu and CentOS
    Switch CD pipeline to CentOS 7 with https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/ This enables us to build with gcc7 C++17 compiler while keeping a relatively old glibc requirement for distribution.
    Simplify ARM Edge builds
        Switch to standard Ubuntu / Debian cross-compilation toolchain for ARMv7, ARMv8
        Switch to https://toolchains.bootlin.com/ toolchain for ARMv6 (the Debian ARMv6 toolchain is for ARMv4 + ARMv5 + ARMv6, but we wish to only target ARMv6 and make use of ARMv6 features)
        Remove reliance on dockcross for cross compilation.
    Simplify Jetson build
        Use standard Ubuntu / Debian cross-compilation toolchain for ARMv8
        Upgrade to Cuda 10 and Jetpack 4.3
        Simplify build setup
    Simplify QEMU ARM virtualization test setup on CI
        Remove complex "Virtual Machine in Docker" logic and run a QEMU based Docker container instead based on arm32v7/ubuntu
    Fix out of bounds vector accesses in
        SoftmaxGradOpType
        MKLDNNFCBackward
    Fix use of non-standard rand_r function (which is not available on anymore on newer Android toolchains and shouldn't be use in any case).
    Fix reproducibility of RNN with Dropout
    Fix reproducibility of DGL Graph Sampling Operators
    Update tests for Android Edge build to NDK19. The previously used standalone toolchain is obsolete.

Those Dockerfiles that required refactoring as part of the effort were refactored based on the following consideration

    Maximize the use of system dependencies provided by the distribution instead of manually installing dependencies from source or from third party vendors. This reduces the complexity of the installation process and essentially pins the dependency versions, increasing CI stability. Further, Dockerfile build speed is improved. To facilitate this, use recent distribution versions. We still ensure backwards compatibility via CentOS7 based build and test stages
    Minimize the number of layers in the Dockerfile. Don't have 5 different script files executed, each calling apt-get update and install, but just execute once. Speeds up the build and reduces image size. Keep each Dockerfile simple and tailored to a purpose, instead of running 20 scripts to install dependencies for every thinkable scenario, which is unmaintainable.

Some more small changes:

    Remove outdated references to Cuda 7 and Cuda 8 in various files.
    Remove C++03 support in mshadow
    Disable broken tests
        NumpyBooleanAssignForwardCPU apache#17990
        test_init.test_rsp_const_init apache#17988
        quantized_elemwise_mul apache#18034

List of squashed commits

* cpp standard

* Remove leftover files of Cuda 7 and Cuda 8 support

* thrust 1.9.8 for clang10

* compiler warnings

* Disable broken test_init.test_rsp_const_init

* Disable tests invoking NumpyBooleanAssignForwardCPU

* Fix out of bounds access in SoftmaxGradOpType

* Use CentOS 7 for staticbuilds

CentOS 7 fullfills the requirements for PEP 599 manylinux-2014 and provides a
C++17 toolchain.

* Fix MKLDNNFCBackward

* Update edge toolchain

* Support platforms without rand_r

* Cleanup random.h

* Greatly simplify qemu setup

* Remove unused functions in Jenkins_steps.groovy

* Skip quantized_elemwise_mul due QuantizedElemwiseMulOpShape bug

* Fix R package installation

apache#18042

* Fix centos ccache

* Fix GPU Makefile staticbuild on CentOS7

* CentOS7 NCCL

* CentOS7 staticbuild fix link with libculibos
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Bug Disabled test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yzhliu @leezu @haojin2