[RFC] Python Array API standardization

[NathanYyc]

Problem statement

Python API Standard is an standard Array API for all Python numerical computing, data science, machine learning, and deep learning frameworks. By specifying an API for the most common ways arrays are constructed and used, It aims to address the differences among those frameworks that makes developers quite difficult to write code that works with multiple (or all) of these libraries.

Proposed solutions

By comparing the Python API Standard and APIs in MXNet.numpy, an list of APIs similar to the list in pytorch Python Array API Compatibility Tracker is formed. Further works will be done according to the list attached. The dlpack synchronization APIs in the list is specified in Specify synchronization semantics.

References

• Python API Standard
• pytorch Python Array API Compatibility Tracker
• dlpack Specify synchronization semantics

On-going Efforts

#20454
#20478

Attachment

MXNet & Python Api Standard Comparison

• current list is base on APIs in MXNet 2.0 & Python Data API Specification(commit: cf5aa666017b8a8a2117aa1cb1770eb1cf841942)
• We classified each API into 6 categories:
  • No Changes: The current PyTorch implementation already conforms to the Data API.
  • New API: Need to add a new function or constant.
  • Enhancement: Need enhancement to the existing function (add new argument, etc.).
  • Enhancement with BC: Need to change signature of the existing function. (breaking change if we implement the API directly under torch.* namespace)
  • Behavior: Need to change behavior of the existing function. (breaking change if we implement the API directly under torch.* namespace)
  • N/A: The detailed specification is not determined yet in the Data API.
• There is 46 New APIs needed, 60 current APIs needed to be enhanced, 0 API needed to change behavior and 2 APIs is not specified in current Data API commit.
  @barry-jin @szha

[github-actions]

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

[AnirudhDagar]

Thanks for opening this discussion. Looking forward to MXNet's support of Array API.
cc @rgommers (for awareness)

[asmeurer]

Hi. Just wanted to comment here to see if there's any way we can help with the mxnet implementation of the array API.

As you may be aware, we have a test suite for array API compliance at https://github.com/data-apis/array-api-tests/. If you have any issues or questions regarding the test suite, please reach out to us by opening an issue in the array-api-tests issue tracker. In particular if you have any ways that you feel the test suite could be easier to use, we'd like to know about that so we can improve things. We want it to be as useful as possible for people implementing array API support in libraries like mxnet.

CC @honno

[honno]

I've written a basic wrapper to get things started, as the test suite requires some things like namespaced dtypes and info functions for most of the tests. I'm quite unfamiliar with MXNet so struggled to get far in making sure the test suite was working fine and all the failures were expected non-compliance issues. https://gist.github.com/honno/c95eeabc71f5ca65d40e5a5105eda672

I think a native and fully-compliant xp.asarray() would be a good priority if folks wanted to start using the test suite, although generally I think it's fair to ignore the test suite until you're much closer to covering the spec.

[barry-jin]

Thanks @asmeurer, @honno. MXNet master branch is using array-api-tests part of test suites in CI (commit hash here). We also have mxnet.numpy.array as standard array API to be interoperable with other array libraries.

[asmeurer]

There have been quite a few additions and general improvements to the test suite since commit c1dba80a196a03f880d2e0a998a272fb3867b720, which is from September. Would it help you if we were to tag releases for the tests repo?

[barry-jin]

Yes, that would be helpful.