Adds PyPI CD Pipeline

[perdasilva]

Description

Updates the CD pipeline to include the PyPI release

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

[perdasilva]

@zachgk could you please take a quick look ^^

[perdasilva]

Here's the pipeline running in dev - I've just started it, so it might take some time to run to completion.

[perdasilva]

Don't merge until #16202 is merged =)

[lanking520]

@szha he is trying to work on the migration, hope if you have some time to take a look.

[szha]

Where can we try to install the wheels? Can we host it in an s3 bucket for trying out first?

[perdasilva]

@szha I can see about either posting them up as artifacts (on dev) or posting them to an S3 bucket - should be fine.

[szha]

@perdasilva great. Let's prepare them and ask people to try them out for verification.

[perdasilva]

Setting this to work in progress until we get community validation of the wheel files

[perdasilva]

Here is a successful run: http://jenkins.mxnet-ci-dev.amazon-ml.com/job/restricted-mxnet-cd/job/mxnet-cd-release-job/236/ - the wheels have been posted up as artifacts.

[perdasilva]

Posted up wheel files and shared them with the community. No negative feedback. Removing WIP and ready to merge =D

[szha]

Would this start publishing to all pypi package variants? As a first step, it would be prudent to limit the blast radius by publishing one package initially.

[perdasilva]

@szha I've added changes to the pypi pipeline to skip publishing all variants except cu101 and cu101mkl - you can see an example here (I only ran the pipeline against cu100, cu101, and cu101mkl to converse resources). I decided to only skip the publish step, so we could still QA the builds for all variants. Once you are comfortable, the filter can be removed.

[aaronmarkham]

I couple of minor consistency suggestions in the readmes.

[aaronmarkham]

Since this is passing CI, I can move my review to Approved and merge (readme's can be updated later). Is it good to go now?

Can we get the Pi and Nano wheels in this pipeline too? I'd love to see that! I have some notes on it and some successes, but when trying to automate the wheel creation I was getting stuck on cross-compile issues in the output wheel's platform designation.

[perdasilva]

@aaronmarkham I've put through some changes to the READMEs. Let's wait for @szha to merge and have final say on the variant I chose as the release one.

Regarding Pi and Nano, eventually it should be possible. I guess this would just mean new variants. So, first, we would need to figure out the compilation, so the binary can be posted up to the artifact repository. Once it's there, we can update this pipeline to include it in the release.

Since we already do some testing in CI, it mightn't be too difficult...

[perdasilva]

@zachgk or anyone, could we please merge this now?

[aaronmarkham]

Looking forward to this!!