[MXNET-908] Enable python tests in Travis

[KellenSunderland]

Description

This turns on python tests in Travis in order to increase or MacOS coverage.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

[kalyc]

Thanks for your contribution @KellenSunderland
I noticed the build is failing
@mxnet-label-bot[pr-awaiting-response]

[KellenSunderland]

The build is failing with the same error users are receiving. It's a valid failure which we were trying to reproduce in CI. The next steps to resolve the problem are to fix the MacOS issues, then merge this PR.

…

On Fri, Sep 14, 2018, 7:48 PM Kalyanee Chendke ***@***.***> wrote: Thanks for your contribution @KellenSunderland <https://github.com/KellenSunderland> I noticed the build is failing @mxnet-label-bot <https://github.com/mxnet-label-bot> [pr-awaiting-response] — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12550 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHGTE41dXviBdT0iIwUwbaiib_VDvVpKks5ua-v0gaJpZM4WnbJC> .

[marcoabreu]

@KellenSunderland it seems like everything is passing. Is this good to merge?

[KellenSunderland]

It's ok to merge. My preference would be to wait a day and then I'll split the fix into it's own PR (to make it easy to revert/bisect/cherry pick into release branches).

…

On Mon, Sep 17, 2018, 9:15 AM Marco de Abreu ***@***.***> wrote: @KellenSunderland <https://github.com/KellenSunderland> it seems like everything is passing. Is this good to merge? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12550 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHGTEyBTgpAFs9lltPM-0C9Fd87WN0ibks5ub6CJgaJpZM4WnbJC> .

[lanking520]

@KellenSunderland instead of placing python here as a job that travis would always execute, could you please put python build section into a separate target so we can also add Scala-maven test in here in the future.

[lanking520]

Possibly this will help: https://docs.travis-ci.com/user/build-stages

[KellenSunderland]

I think it would be great if we could add Scala tests as well, but I do have a few concerns. How long would they take to execute? Let's keep in mind that we have a limited number of executors that can be run before we begin interfering with other Apache projects. There's also a global timeout of ~45 minutes to build and execute on a virtualized MacOS host which takes a few minutes just to start. I'd ideally like to keep our total test time below 30 minutes to avoid hitting this in the future.

In principle I'm not against build stages, but unless we have a lot of jobs running in parallel that have dependencies on each other I don't see a big benefit in using them.

[lanking520]

Thanks for your reply! If you mean scalaunittestit will be around 10 mins or less. Is travis wait possible to apply in here?

[KellenSunderland]

Unfortunately Travis wait won't help in this case. The tests are active and sending output to Travis, they just take a long time and hit the global timeout for Travis jobs.

10 minutes actually is reasonable. Let's do a test run with Scala enabled after this is merged and see if we can get it included. My guess is if we enable folder caching, and cache models we're currently downloading we'll be able to fit it in and not hit the global timeouts.

[lanking520]

Sure, let's try to make Scala test available later in Travis.

[KellenSunderland]

Note: this will fail in Travis until this PR is merged: #12590

[KellenSunderland]

Rebased, should build properly now.

[lanking520]

Looks like we are still suffering the timeout issue: https://travis-ci.org/apache/incubator-mxnet/builds/433814087?utm_source=github_status&utm_medium=notification