This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Large Index Support:Part 1 #15728

Closed

access2rohit wants to merge 6 commits into apache:master from access2rohit:lis_slice_6

Contributor

access2rohit commented Aug 2, 2019 •

edited

Loading

Description

New int64 C APIs.
Changes related to using index_t in slice operator.
New Test file for testing very large indices

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Large Index Support APIs Part1

access2rohit requested review from anirudh2290, eric-haibin-lin and szha as code owners

August 2, 2019 00:16

Rohit Kumar Srivastava added 5 commits

August 2, 2019 16:40


          Adding Large Index Support for slice operator

56da161


          adding changes to fix py2 related error in CI/CD

139aa8e


          revert unit32 to unsigned int

fa402c2

This reverts commit c6ee19f.


          fixing base.py

1999b7b


          reveting changes with respect to array type 'q' in python

15572d1

access2rohit force-pushed the lis_slice_6 branch from dc5313d to 15572d1 Compare

August 2, 2019 16:40

access2rohit changed the title ~~[WIP]LIS slice 6~~ Large Index Support:Part 1

Contributor

piyushghai commented Aug 2, 2019

@mxnet-label-bot Add [Backend, pr-awaiting-review]

marcoabreu added Backend pr-awaiting-review labels

apeforest reviewed

View reviewed changes

src/c_api/c_api.cc

@@ @@ -188,13 +188,29 @@ int MXNDArrayCreate(const mx_uint *shape, @@
                 API_END();
               }
+              int MXNDArrayCreateExInt64(const mx_int64 *shape,

Contributor

apeforest Aug 2, 2019

If the internal implementation is the same and only differs by data type, we can create an internal template to do this?

apeforest reviewed

View reviewed changes

src/c_api/c_api.cc

@@ @@ -550,6 +566,34 @@ int MXNDArrayGetShapeEx(NDArrayHandle handle, @@
                 API_END();
               }
+              int MXNDArrayGetShapeExInt64(NDArrayHandle handle,

Contributor

apeforest Aug 2, 2019

If the internal implementation is the same and only differs by data type, we can create an internal template to do this?

apeforest reviewed

View reviewed changes

src/c_api/c_api_common.h

@@ @@ -130,6 +134,30 @@ struct MXAPIThreadLocalEntry { @@
                     }
                   }
                 }
+                inline static void SetupShapeArrayReturnWithBufferExInt64(

Contributor

apeforest Aug 2, 2019

If the internal implementation is the same and only differs by data type, we can create an internal template to do this?

apeforest reviewed

View reviewed changes

src/c_api/c_api_symbolic.cc

@@ @@ -658,6 +658,85 @@ int MXSymbolInferShapeEx(SymbolHandle sym, @@
                 API_END();
               }
+              int MXSymbolInferShapeExInt64(SymbolHandle sym,

Contributor

apeforest Aug 2, 2019

If the internal implementation is the same and only differs by data type, we can create an internal template to do this?

apeforest reviewed

View reviewed changes

src/c_api/c_api_symbolic.cc

@@ @@ -708,6 +787,31 @@ int MXSymbolInferShapePartialEx(SymbolHandle sym, @@
                                             &succ);
               }
+              int MXSymbolInferShapePartialExInt64(SymbolHandle sym,

Contributor

apeforest Aug 2, 2019

If the internal implementation is the same and only differs by data type, we can create an internal template to do this?

apeforest reviewed

View reviewed changes

src/c_api/c_api_symbolic.cc

+                                                   const int **aux_shape_ndim,
+                                                   const int64_t ***aux_shape_data,
+                                                   int *complete) {
+                int succ;

Contributor

apeforest Aug 2, 2019

initilize the value of succ and better to rename it success

apeforest reviewed

View reviewed changes

tests/nightly/test_large_vector.py

		@@ -0,0 +1,37 @@
		# Licensed to the Apache Software Foundation (ASF) under one

Contributor

apeforest Aug 2, 2019

Will this test pass with your Part-1 PR? If not, please check it in together with your Part-2 without causing nightly test failure.

apeforest reviewed

View reviewed changes

python/mxnet/ndarray/ndarray.py Outdated

-                      pdata = ctypes.POINTER(mx_int)()
-                      check_call(_LIB.MXNDArrayGetShapeEx(
-                          self.handle, ctypes.byref(ndim), ctypes.byref(pdata)))
+                      if Features().is_enabled('INT64_TENSOR_SIZE') and sys.version_info[0] > 2:

Contributor

apeforest Aug 2, 2019

I think calling the Feature().is_enabled() may take extra time here


          rearrange system call and slower Feature() call

3bfd375

Contributor Author

access2rohit commented Aug 17, 2019

Feature already merged in: #15593

access2rohit closed this

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

apeforest apeforest left review comments

anirudh2290 Awaiting requested review from anirudh2290

eric-haibin-lin Awaiting requested review from eric-haibin-lin eric-haibin-lin is a code owner

szha Awaiting requested review from szha szha is a code owner

Labels

Backend pr-awaiting-review