LoadError: Cannot find or load libmxnet.dylib

[conradwt]

Hi, I have performed the following steps:

1). built MXNet from source with GPU support which generated the following 2 files:

libmxnet.a
libmxnet.so

2). withing Julia 0.6.2, Pkg.add("MXNet")

julia> Pkg.add("MXNet")
INFO: Installing MXNet v0.3.0
INFO: Building MXNet
INFO: MXNET_HOME environment detected: /Users/conradwt/ai.dir/projects/mxnet
INFO: Trying to load existing libmxnet...
INFO: Failed to load existing libmxnet, trying to build from source...
INFO: Found a CUDA installation.
INFO: Found a CuDNN installation.
INFO: CUDA_HOME -> /usr/local/cuda
INFO: Unsure if we can build against mkl.
INFO: Attempting build anyway.
INFO: USE_JULIA_BLAS -> true
INFO: Package database updated

3). within Julia 0.6.2, using MXNet

julia> using MXNet
INFO: Precompiling module MXNet.
ERROR: LoadError: LoadError: Cannot find or load libmxnet.dylib. Please see the document on how to build it.
Stacktrace:
 [1] error(::String) at ./error.jl:21
 [2] include_from_node1(::String) at ./loading.jl:576
 [3] include(::String) at ./sysimg.jl:14
 [4] include_from_node1(::String) at ./loading.jl:576
 [5] include(::String) at ./sysimg.jl:14
 [6] anonymous at ./<missing>:2
while loading /Applications/JuliaPro-0.6.2.2.app/Contents/Resources/pkgs-0.6.2.2/v0.6/MXNet/src/base.jl, in expression starting on line 38
while loading /Applications/JuliaPro-0.6.2.2.app/Contents/Resources/pkgs-0.6.2.2/v0.6/MXNet/src/MXNet.jl, in expression starting on line 25
ERROR: Failed to precompile MXNet to /Applications/JuliaPro-0.6.2.2.app/Contents/Resources/pkgs-0.6.2.2/lib/v0.6/MXNet.ji.
Stacktrace:
 [1] compilecache(::String) at ./loading.jl:710
 [2] _require(::Symbol) at ./loading.jl:497
 [3] require(::Symbol) at ./loading.jl:405

Note: libmxnet.dylib was not created but a libmxnet.so and libmxnet.a were created in step (1). Thus, I'm not sure why using MXNet is looking for a different file here.

Comment

Platform

• macOS 10.13.3
• Julia 0.6.2
• MXNet 1.1.0

[iblislin]

The build script will copy and rename it as libmxnet.dylib if build from Pkg.build.

MXNet.jl/deps/build.jl

Line 255 in 3df1214

`cp $_mxdir/lib/libmxnet.so $_libdir/libmxnet.$(Libdl.dlext)`

Did you build your libmxnet manually? or via Pkg.build?

[iblislin]

Also, the stable release of MXNet.jl might not work together with newer libmxnet.
If you ran into issues, please try out master of MXNet.jl first.
Lots of chages from stable release have been listed in NEWS.md.

[conradwt]

I’m manually installing libmxnet from source from the project’s web site but I’ll try installing MXNet.jl using Pkg.build.

[conradwt]

@iblis17 Please advise how to build MXNet.lj using Pkg.build. From the Julia documentation, there's no option for a single package. Finally, I believe part of the issue which I missed earlier is that Pkg.add("MXNet") is having trouble locating the package libmxnet.so.

[iblislin]

In your case, I think adding a symblink (or just copy) libmxnet.so to libmxnet.dylib may work.

Please advise how to build MXNet.lj using Pkg.build.

try Pkg.build("MXNet")?

Finally, I believe part of the issue which I missed earlier is that Pkg.add("MXNet") is having trouble locating the package libmxnet.so.

yeah, I add libmxnet.so into search list for build script.
Could you check out #424 ?

[conradwt]

I was able to resolve the Pkg.add("MXNet") by performing the following actions:

• create a relative path within the libmxnet.so

$ install_name_tool -id @rpath/libmxnet.so libmxnet.so

• create a symbolic link

$ ln -s libmxnet.so libmxnet.dylib

• installing the package

julia> Pkg.add("MXNet")
INFO: Installing MXNet v0.3.0
INFO: Installing TakingBroadcastSeriously v0.1.1
INFO: Building MXNet
INFO: MXNET_HOME environment detected: /Users/conradwt/ai.dir/projects/mxnet
INFO: Trying to load existing libmxnet...
INFO: Existing libmxnet detected at /Users/conradwt/ai.dir/projects/mxnet/lib/libmxnet.dylib, skip building...
INFO: Found a CUDA installation.
INFO: Found a CuDNN installation.
INFO: CUDA_HOME -> /usr/local/cuda
INFO: Package database updated
INFO: METADATA is out-of-date — you may not have the latest version of MXNet
INFO: Use `Pkg.update()` to get the latest versions of your packages

• using the package

julia> using MXNet
INFO: Recompiling stale cache file /Applications/JuliaPro-0.6.2.2.app/Contents/Resources/pkgs-0.6.2.2/lib/v0.6/MXNet.ji for module MXNet.

Using the install_name_tool and creating the symbolic link would be unnecessary if the appropriate build rules for macOS were added to Apache MXNet.

[iblislin]

no idea why you need install_name_tool, in our Travis CI build, we simply do copy.

[iblislin]

Does #424 works for you if not to make any symlink?

[conradwt]

Adding just the sym-link didn’t work for me because this would generate a failure when I tried Pkg.add(“MXNet”). Why? The relative path, lib/libmxnet.so, within libmxnet.so isn’t correct. @rpath/libmxnet.so is a much better way to specify a relative path for shared and dynamic libraries.

[iblislin]

Adding just the sym-link didn’t work for me because this would generate a failure when I tried Pkg.add(“MXNet”). Why? The relative path, lib/libmxnet.so, within libmxnet.so isn’t correct. .so is a much better way to specify a relative path for shared and dynamic libraries.

Sorry, I do not get detail issue you described. But I mean, don't make any symlink. Can just copying work for you?

[conradwt]

If I don’t make a symlink, then I’ll get a failure.

[iblislin]

Is it still a LoadError? mind share the error message?

or...could you try that copying your libmxnet.so to Pkg.dir("MXNet", "deps", "usr", "lib"), then rename it as libmxnet.dylib?

[conradwt]

The error that I'm getting appear in my initial message which is as follows:

julia> Pkg.add("MXNet")
INFO: Installing MXNet v0.3.0
INFO: Building MXNet
INFO: MXNET_HOME environment detected: /Users/conradwt/ai.dir/projects/mxnet
INFO: Trying to load existing libmxnet...
INFO: Failed to load existing libmxnet, trying to build from source...
INFO: Found a CUDA installation.
INFO: Found a CuDNN installation.
INFO: CUDA_HOME -> /usr/local/cuda
INFO: Unsure if we can build against mkl.
INFO: Attempting build anyway.
INFO: USE_JULIA_BLAS -> true
INFO: Package database updated

[iblislin]

Oh, I found the problem in my description. What I expected is this:

1. you already finished Pkg.add("MXNet")
2. cd to Pkg.dir("MXNet", "deps", "usr", "lib") (create this dir if not exists)
3. cp /path/to/your/libmxnet.so ./libmxnet.dylib
4. then, issue using MXNet in you REPL and check the precompilation success or not.

---

If you want stuffs work out of box from Pkg.add("MXNet") or Pkg.build("MXNet"),
you should apply this patch first: https://github.com/dmlc/MXNet.jl/pull/424/files (hope this patch solve this Pkg.add issue).

Instructions:
1.

cd `julia -E 'Pkg.dir("MXNet")'`

2. git fetch origin pull/424/head:pr-424
3. git checkout pr-424
4. set your MXNET_HOME
5. Pkg.build("MXNet")
6. using MXNet

[conradwt]

Here's the current status of things:

• I copied the libmxnet.so to Pkg.dir("MXNet", "deps", "usr", "lib") and renamed it as libmxnet.dylib

• downloaded this repo and switch to the v0.3.0 branch

• julia lenet.jl ran successfully on the GPU.

Now, things appear to be working now. So, does the master branch have this patch? If so, how does one simply use the master branch? From the Julia documentation, one should be able to do the following: Pkg.checkout("MXNet"). However, this doesn't work as expected.

[iblislin]

Now, things appear to be working now. So, does the master branch have this patch?

well, there are two situations:

1. without patch, just copying, and issue using MXNet.
2. applying patch, and run Pkg.build, using MXNet.

According to your operation in #423 (comment).
You chose approach 1 and it works.

So, does the master branch have this patch?

No, I sent it as a PR, not merged yet.

If so, how does one simply use the master branch? From the Julia documentation, one should be able to do the following: Pkg.checkout("MXNet"). However, this doesn't work as expected.

Pkg.checkout("MXNet") then Pkg.build("MXNet").
But note that master branch doesn't contain #424 .