[WIP][MXNET-1302,MXNET-1331] Backport fix to v1.3.x branch

[jessr92]

Consumers of MXNet are exposed to the commons-codec and commons-io library versions MXNet uses internally. This change excludes those JARs from the JARs created with make scalapkg.

Description

This PR removes commons-codec and commons-io from the assembled JAR and adds both packages as dependencies in deploy.xml

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• [] Changes are complete (i.e. I finished coding on this PR)
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• 3rd party libraries removed from assembled JAR
• [] 3rd party libraries added as dependencies to deployed POM

Comments

Backport of pull request 14000 and 14303 to v1.3.x branch

Originally reported in #13929

[jessr92]

@mxnet-label-bot update [maven, scala, pr-work-in-progress]

[ChaiBapchya]

Thanks for your contribution.

Looks like it is failing on windows build. All 3 failures are related to same line

File "ci/build_windows.py", line 33, in <module>

    from util import *

  File "C:\jenkins_slave\workspace\build-gpu\ci\util.py", line 21

    def get_mxnet_root() -> str:

Have a look.

[lanking520]

@yajiedesign Could you please take a look if you know what has been changed on master?

[jessr92]

Given the error, looks like the workflow migrated from Python 2 to Python 3? I think it's complaining about the "arrow" syntax.

[zachgk]

@perdasilva Can you help take a look at this? It seems to be failing on the Windows CI

[perdasilva]

@zachgk right. There've been changes to the windows ci pipeline. I'll need to back port the changes to v1.3.x. I'll make that PR today for you and try to get it merged asap. The PR to bring these changes over: #14469

[lanking520]

@gordon1992 Got @gordon1992 PR in. Please try to pull --rebase with 1.3.x to pass the CI. Thanks!

[jessr92]

Apologise for the delay, I've been on annual leave. I have now rebased.

[jessr92]

Looks like the new failure is to do with my exclusion of Scala modules. This makes sense since this PR is WIP with the remaining task to declare dependencies on the excluded modules. Expectation is that the build will pass after that has been completed.

[lanking520]

It seemed like Clojure failed because of that

[zachgk]

@gordon1992, it might be easier to test this locally. What you need to do is:

1. Build Engine following https://mxnet.incubator.apache.org/versions/master/install/scala_setup.html
2. Run mvn install from the scala-package folder to build Scala and load it into your maven cache
3. Install the clojure tool following https://github.com/technomancy/leiningen/wiki/Packaging
4. Run lein test from the contrib/clojure-package folder to run the Clojure tests (pulling the Scala jars from your cache).

[karan6181]

@gordon1992 Could you please try the suggestion provided by @zachgk

[jessr92]

Hello,

Apologies for the really bad delays to working on this. Trying to set time aside for this has been tricky.

From what I can see, backporting the fix in a clean way is tricky given the Maven build restructuring that has occurred (and is only present in 1.5 onwards).

Thanks for your patience,
Gordon

[jessr92]

Unfortunately I'm unlikely to get to this. Sorry.