RFE: Single-JAR snapshots/releases

[kkofler]

Currently, all your snapshots and releases are distributed through Maven. This is great for developers that are already using Maven, but it is very impractical if one wants to use Netlib-Java in projects using other build systems, or upgrade the version bundled by third-party projects. (In our case, we would like to make several existing projects, some we build using NetBeans, some third-party ones, take advantage of the new Netlib-Java snapshots and their revamped support for native LAPACK/BLAS, including for Windows, which the old version did not support.)

It would be nice if you could generate single JARs for the snapshots, which would contain the core and all the native libraries (.so/.dll/.dylib) in one JAR (the code will figure out the correct one to pick based on the platform anyway, won't it?). (For Windows, it would probably make sense to also bundle the OpenBLAS DLLs, if that's what you build native-system against on Windows. For GNU/Linux, I'd expect native-system to really use what's on the system (provided by the distro), of course.)

[fommil]

@kkofler sorry, but I'm not going to fix this:

1. it would complicate (the already very complicated) build/release process on my side
2. it would mean that there are three copies of each native file in central maven... I'm embarrassed enough that there are two.
3. if people choose to step outside the maven / ivy world then they bring this onto themselves.
4. bundling "system optimized" natives is against the whole point of that backend. It is the responsibility of the end user / provisioning to ensure that they are available at runtime. I'm not going to be distributing one vendor's generic implementation: that's what the native ref is for.

[fommil]

@kkofler as a workaround, you can build your own single jar distro. just look in the perf module for how to do that.

Specifically

cd perf
mvn compile assembly:single

https://github.com/fommil/netlib-java/blob/master/perf/pom.xml#L9

although you might want to copy the pom.xml to your own filesystem, delete the src and remove the dependencies that are only for the performance tests (e.g. Guava, OpenCSV).

[fommil]

@kkofler another workaround is to build a single MTJ and netlib-java jar, the same assembly command will work with the trunk https://github.com/fommil/matrix-toolkits-java/

[kkofler]

Uhm, mvn compile assembly:single in perf drags in a lot of unwanted dependencies (which are used in the benchmarking process). I created this netlib-java/pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!-- mvn compile assembly:single -->

    <parent>
        <groupId>com.github.fommil.netlib</groupId>
        <artifactId>parent</artifactId>
        <version>1.1-SNAPSHOT</version>
    </parent>

    <artifactId>netlib-java</artifactId>
    <packaging>jar</packaging>

    <dependencies>
        <dependency>
            <groupId>${project.parent.groupId}</groupId>
            <version>${project.parent.version}</version>
            <artifactId>all</artifactId>
            <type>pom</type>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>${exec.mainClass}</mainClass>
                        </manifest>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

which seems to do more or less what I want (but yikes, those natives are huge! :-( ). Wouldn't that be easily scripted into your build process? Or are you doing everything by hand?

But a remaining issue is that I'd really like to also get it to include the legacy directory, and somehow I can't get that to work. Adding the dependency:

        <dependency>  <!-- legacy API backwards compatibility-->
            <groupId>com.googlecode.netlib-java</groupId>
            <version>${project.parent.version}</version>
            <artifactId>netlib-java</artifactId>
            <type>pom</type>
        </dependency>

the obvious way does nothing (and I tried changing my artifactId to something different from "netlib-java", that didn't help either).

As for:

bundling "system optimized" natives is against the whole point of that backend. It is the responsibility of the
end user / provisioning to ensure that they are available at runtime. I'm not going to be distributing one
vendor's generic implementation: that's what the native ref is for.

FYI, OpenBLAS's "generic" builds are not unoptimized builds like the reference ones, they do runtime CPU detection and automatically select the optimized assembly kernels for the CPU detected at runtime. There are also no licensing issues with distributing them. (It's BSD-licensed.) That said, I now agree that -system is not the right backend for that. What would you think of a native-openblas backend (which would also look for the libopenblas they ship rather than for libblas and liblapack, or maybe link it statically like native-ref does)? I think many developers would be happy with shipping that (and only that, omitting native-ref which would be slower and native-system which is too complicated to set up for most end users), if it were available.

[fommil]

@kkofler remove the <type>pom</type> from the legacy API. This will be removed in the next release, so I suggest you migrate now. Yes, I noted that you should remove the performance test dependencies.

Yes, the binaries are rather large, hence my reluctance to distribute an additional file that doubles the storage and bandwidth requirements. I also have absolutely no intention of encouraging self management of jars. The correct solution is for you to use a dependency manager: use Ivy if you're stuck with ant (the legacy build system of Netbeans).

The OpenBLAS builds are generic as they don't include a lot of the compiler optimisation flags that one would expect of a system optimised binary. They are also generically tuned instead of machine empirically tuned. The instructions for enabling OpenBLAS on a Windows machine are in the README and please encourage OpenBLAS to build statically against the gcc / gfortran libs to make the installation even easier for end users and system admins: OpenMathLib/OpenBLAS#296 Installation amounts to dropping a file into a library, it doesn't get much simpler than that.

I have no intentions of distributing vendor specific implementations. They either implement BLAS/LAPACK or they don't. If they do, NativeSystem is how to use it.