commons-codec/io part of make scalapkg output JAR

[jessr92]

This is related #13528 (which is referring to the inclusion of the Scala standard library).

The JAR output by make scalapkg (say mxnet-full_2.11-INTERNAL.jar) contains commons-codec (1.10) and commons-io (2.1).

This is causing classpath issues when depending on MXNet and commons-io directly.

In my specific case I depend on commons-io 2.5 but the commons-io 2.1 bundled by MXNet is shadowing my own dependency on 2.5.

I have a commit in my fork which excludes commons-codec and commons-io from the assembly jessr92@0b141f7 but I would like to assist in fixing the problem directly.

I see two options:

1. Explicitly exclude commons-codec and commons-io from the assembly. The MXNet library then needs to add commons-codec and commons-io as a transitive dependency.

2. Shadow the commons-codec and commons-io library using Maven Shade Plugin: https://maven.apache.org/plugins/maven-shade-plugin/

My current knowledge of Maven is sufficiently shallow that I would appreciate input from maintainers regarding the most appropriate fix for the project and making the fix more general (removal of Scala standard library, SLF4j and any other 3rd party libraries currently bundled).

Thanks

[piyushghai]

Hi @gordon1992 Thanks for posting this issue. @frankfliu Can you have a look here regarding the bundling of commons-codec and commons-io in the jar?

@mxnet-label-bot add [Scala]

[frankfliu]

@gordon1992
Thank you for looking into this issue. Your fix isn't a complete fix:

1. There are other dependencies will have the same issue.
2. The deployed pom.xml didn't declare those dependencies. Developer must manually add those dependencies in their project.

We will take a look and see what we can do to fix it.

[jessr92]

Thanks for the quick response.

Yeah, the fix was sufficient for my investigation.

Completely agree on the next two points. Thanks for looking into it.