[TUTORIAL] Gluon performance tips and tricks

[thomelane]

Summary of performance features provided in MXNet that are relevant for Gluon users. Useful for a beginner audience who might not know what features to search for in the first place.

Description

(Brief description on what this PR is about)

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[thomelane]

@aaronmarkham would be great if you could review this new tutorial too, cheers! i'll make adjustments to my sparse tutorial tomorrow.

[pengzhao-intel]

It's great to have these documents. I think we still need to cover CPU performance.
@xinyu-intel will help to provide additional information about CPU

[thomelane]

Would be great to get CPU specific tricks and tips @pengzhao-intel. Given the length of this tutorial already let's add those in another tutorial as a follow up. @xinyu-intel are you able to write about some performance tricks for CPU and follow up in another PR? When we have that we can think about splitting this up into generic tips, GPU specific and CPU specific tricks.

[pengzhao-intel]

Would be great to get CPU specific tricks and tips @pengzhao-intel. Given the length of this tutorial already let's add those in another tutorial as a follow up. @xinyu-intel are you able to write about some performance tricks for CPU and follow up in another PR? When we have that we can think about splitting this up into generic tips, GPU specific and CPU specific tricks.

Agree. I see lots of tips and tricks are common for both CPU and GPU; so we can make the user aware of this :)

[aaronmarkham]

Couple of minor things... otherwise looks good.

[aaronmarkham]

I restarted the failed CI test for you.

[thomelane]

thanks for all the reviews @aaronmarkham @pengzhao-intel @ptrendx. updates have been made.

[pengzhao-intel]

Prebuild link:

http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-15427/4/tutorials/gluon/performance.html

[pengzhao-intel]

It's a nice tutorial. Add minor comments.

After going through these approaches, how people know that they have already got a good performance?

[thomelane]

@pengzhao-intel thanks for the follow-up. made the changes.

[pengzhao-intel]

Thanks for the nice document.

[wkcn]

Great Work! Thank you : )

[braindotai]

Marvellous work by @thomelane, hands down.
Thanks a gazillion times.

[braindotai]

A good tutorial on how to convert a Gluon model to Module and vice-versa would be great. I've seen many issues and confusions on Github and MXNet-Forum regarding it. Thanks