CodecZLib build fails on Linux and WSL #170

Crown421 · 2019-07-04T00:53:49Z

Dear Team,

I am trying to setup a new server with Julia, and while trying to build JLD2 the build of CodecZlib fails.
The error message is the following:

┌ Error: Error building `CodecZlib`:
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/homes/sridderb/.julia/packages/CodecZlib/9jDi1/deps/usr)")
is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#165(::Bool, ::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /homes/sridderb/.julia/packages/B
inaryProvider/A0sDa/src/Products.jl:419
│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_fi
le), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [4] top-level scope at /homes/sridderb/.julia/packages/CodecZlib/9jDi1/deps/build.jl:93
│  [5] include at ./boot.jl:326 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1038
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:403
│  [9] top-level scope at none:0
    end
│ in expression starting at /homes/sridderb/.julia/packages/CodecZlib/9jDi1/deps/build.jl:78
│ LibraryProduct[LibraryProduct(nothing, ["libz"], :libz, "Prefix(/homes/sridderb/.julia/packages/CodecZlib/9jDi1/deps/usr)")]

Digging through the source code of CodecZLib's build.jl I find that on line 7 they define

products = [
    LibraryProduct(prefix, ["libz"], :libz),
]

which is at odds with the error above, where appear to be 4 inputs, and the prefix is at the end.
Further in line 76

unsatisfied = any(!satisfied(p; verbose=verbose) for p in products)

clearly ends with unsatisfied = false, as we progress to line 96,

write_deps_file(joinpath(@__DIR__, "deps.jl"), products, verbose=verbose)

Now, in BinaryProvider's product.jl we have at line 419 (as mentioned by the error)

for p in products
        if !satisfied(p; verbose=verbose, isolate=isolate)
            error("$p is not satisfied, cannot generate deps.jl!")
        end
end

which looks almost identical to the line above, which goes through.

Further, on my WSL setup, where I ~~have~~ had working JLD2 and CodecZLib, rebuilding it now fails with the same error as above.

I am not sure what to do about this, as I am not highly proficient in Julia yet, but I would appreciate assistance.

Many thanks.

Update
After (painfully) downgrading to version 0.5.4 it goes through without any issue, so there appears to be a bug in the latest release.

The text was updated successfully, but these errors were encountered:

BoundaryValueProblems · 2019-07-04T18:28:56Z

With BinaryProvider v0.5.5, I have the same build problems on all of my machines (macOS, Windows 10, ubuntu/linux), not only in CodecZlib, but also, LibCURL, EzXML, MbedTLS, ZMQ, FreeType. Under v0.5.4, all of them were working without any problem. Please fix this ASAP!!
The error messages for those packages are in the following format:

ERROR: LoadError: LibraryProduct(nothing, ["libfreetype"], :libfreetype, "Prefix(/Users/xxx/.julia/packages/FreeType/2dE5w/deps/usr)") is not satisfied, cannot generate deps.jl!

zgornel · 2019-07-05T09:55:41Z

The error seems to be here

BinaryProvider.jl/src/Products.jl

Line 398 in 6779f54

verbose::Bool=false, isolate::Bool=true) where {P <: Product}

isolate=true triggers the error (at least on my Linux system) and this may be related to

BinaryProvider.jl/src/Products.jl

Line 168 in 6779f54

dl_esc_path = replace(dl_path, "\\"=>"\\\\")

which was introduced in v0.5.5. It looks like a Windows-only fix of some sort that breaks on other systems.

This can be easily fixed I suppose and a new bugfix release done.
@hhaensel Should be the most informed on the matter...

hhaensel · 2019-07-05T11:28:29Z

Hi, could you try to comment out the line 168 and see whether that is the root cause?
If so, could you please report the value of dl_path

- dl_esc_path = replace(dl_path, "\\"=>"\\\\")
+ dl_esc_path = dl_path
+ @info(dl_path)

UPDATE:
on my Windows 7 system, I obtain:

(v1.1) pkg> build -v JLD2
  Building CodecZlib → `C:\Users\m136270\.julia\packages\CodecZlib\9jDi1\deps\build.log`
 Resolving package versions...
[ Info: dl_esc_path: C:\\Users\\m136270\\.julia\\packages\\CodecZlib\\9jDi1\\deps\\usr\\bin\\libz-1.2.11.dll

whithout modification of dl_esc_path and

@info("dl_esc_path: $(dl_esc_path)")

zgornel · 2019-07-05T12:28:48Z

Hmm, it is not that
The output in both cases is (with/without the replace call):

(v1.1) pkg> build -v JLD2
  Building CodecZlib → `~/.julia/packages/CodecZlib/9jDi1/deps/build.log`
[ Info: /home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/usr/lib/libz.so
[ Info: /home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/usr/lib/libz.so.1
[ Info: /home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/usr/lib/libz.so.1.2.11
ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
 [1] #write_deps_file#165(::Bool, ::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/zgornel/.julia/packages/BinaryProvider/A0sDa/src/Products.jl:421
 [2] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
 [3] top-level scope at /home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/build.jl:93
 [4] include(::String) at ./client.jl:403
 [5] top-level scope at none:0
in expression starting at /home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/build.jl:78
┌ Error: Error building `CodecZlib`:
└ @ Pkg.Operations /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075

However, I can confirm that setting isolate=false as default in write_deps_file (L398) works.

Tried the following:

dl_esc_path = replace(dl_path, "\\"=>"\\\\")
if success(`$(Base.julia_cmd()) -e "import Libdl; Libdl.dlopen(\"$(dl_esc_path)\")"`)
    return dl_path
else
    println("failed for $dl_esc_path")
    return dl_path
end

And the output is

v1.1) pkg> build -v JLD2
  Building CodecZlib → `~/.julia/packages/CodecZlib/9jDi1/deps/build.log`
failed for /home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/usr/lib/libz.so

However:

dl_esc_path = "/home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/usr/lib/libz.so"
 success(`$(Base.julia_cmd()) -e "import Libdl; Libdl.dlopen(\"$(dl_esc_path)\")"`)

returns true

hhaensel · 2019-07-05T12:31:41Z

Ah, then it is probably the usage of the quotes somehow

hhaensel · 2019-07-05T12:33:37Z

Please execute

`$(Base.julia_cmd()) -e "import Libdl; Libdl.dlopen(\"test\")"`

and tell me the response

zgornel · 2019-07-05T12:34:57Z

Here it is:

`/usr/bin/julia -Cnative -J/usr/lib/julia/sys.so -g1 -e 'import Libdl; Libdl.dlopen("test")'`

hhaensel · 2019-07-05T12:45:25Z

Well, I don't have an immediate clue ...

hhaensel · 2019-07-05T12:54:49Z

Could you provide line 93 of your
/home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/build.jl

Well, probably better send the whole file

hhaensel · 2019-07-05T13:16:18Z

If I remove te replacement "\\"=>"\\\\" on Windows I receive very similar output as you

(v1.1) pkg> build -v JLD2
  Building CodecZlib → `C:\Users\hh\.julia\packages\CodecZlib\9jDi1\deps\build.log`
 Resolving package versions...
[ Info: dl_esc_path: C:\Users\hh\.julia\packages\CodecZlib\9jDi1\deps\usr\bin\libz-1.2.11.dll
[ Info: dl_esc_path: C:\Users\hh\.julia\packages\CodecZlib\9jDi1\deps\usr\bin\libz-1.dll
[ Info: dl_esc_path: C:\Users\hh\.julia\packages\CodecZlib\9jDi1\deps\usr\bin\libz.dll
ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(C:\\Users\\hh\\.julia\\packages\\CodecZlib\\9jDi1\\deps\\usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] #write_deps_file#169(::Bool, ::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at C:\Users\hh\.julia\dev\BinaryProvider\src\Products.jl:420
 [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at .\none:0
 [4] top-level scope at C:\Users\hh\.julia\packages\CodecZlib\9jDi1\deps\build.jl:93
 [5] include at .\boot.jl:326 [inlined]
 [6] include_relative(::Module, ::String) at .\loading.jl:1038
 [7] include(::Module, ::String) at .\sysimg.jl:29
 [8] include(::String) at .\client.jl:403
 [9] top-level scope at none:0
in expression starting at C:\Users\hh\.julia\packages\CodecZlib\9jDi1\deps\build.jl:78
┌ Error: Error building `CodecZlib`:
└ @ Pkg.Operations C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Pkg\src\Operations.jl:1075

So I think it comes from the fact that the system does not find the file.
I assume it is a symlink?

hhaensel · 2019-07-05T13:30:01Z

In that case you could try to replace the symlink with a copy of the original file and retest.
Alternatively you could try to create an environment vaiable BINARYPROVIDER_COPYDEREF and set it to true and rerun the installation.

hhaensel · 2019-07-05T13:31:54Z

... to me it seems that you have different access rights in your REPL and in the run($(Base.julia_cmd()))

... I'll be offline for a while

zgornel · 2019-07-05T15:05:52Z

@hhaensel , indeed, it looks like a permission problem as I cannot reproduce the problem on a second machine.
So,

libz.so is a symlink indeed, lilke libz.so.1 (both point to the same file)
the ENV["BINARYPROVIDER_COPYDEREF"]=true or copying the lib in place of the symlink do not work
wiping out the .julia directory solves the problem
I do not see any way of tracing this issue back to something specific so far
It may be that the problem is not related to BinaryProvider. Thanks for the assistance.

hhaensel · 2019-07-05T22:45:00Z

We came across a similar issue recently, when ImageMagick could not be rebuild without removing the binaries beforehand: ImageMagick, PR 159 and FileIO, Issue 229
And I realised in the discussion of PR 166 that - similar to @zgornel 's remark above - the first check with satisfied involves default isolate=false whereas write_deps_file involves default isolate=true.
@staticfloat , maybe it would be good to establish some kind of version checking for binaries?
Alternatively, one could think about a force option for build, so that installation is dependent on satisfied(...) || force = true?

hhaensel · 2019-07-05T22:53:36Z

@hhaensel , indeed, it looks like a permission problem as I cannot reproduce the problem on a second machine.
So,

* `libz.so` is a symlink indeed, lilke `libz.so.1` (both point to the same file)

* the `ENV["BINARYPROVIDER_COPYDEREF"]=false` or copying the lib in place of the symlink do not work

* wiping out the `.julia` directory **solves the problem**

* I do not see any way of tracing this issue back to something specific so far
  It may be that the problem is not related to `BinaryProvider`. Thanks for the assistance.

It should have been ENV["BINARYPROVIDER_COPYDEREF"]=true ;-)
But if you found that replacing the symlinks by hardcopies did not help the error has probably been elsewhere...
It would be good to hear from @Crown421 and @BoundaryValueProblems whether their problems can be solved by removing the deps folder.

BoundaryValueProblems · 2019-07-05T23:13:35Z

@hhaensel, I just tried FreeType.jl by removing deps folder and building it via (v1.1) pkg> build FreeType. Nothing happened and deps was not created.
On the other hand if I go back to BinaryProvider v0.5.4, every single packages is built without any problem. So, for a while, I have to stick with v0.5.4 until v0.5.5 is fixed.

hhaensel · 2019-07-05T23:16:58Z

Two questions:

Did the error message change?
Is removing (or temporarily renaming) the whole .julia folder an option as @zgornel wrote?

staticfloat · 2019-07-06T01:07:20Z

@staticfloat , maybe it would be good to establish some kind of version checking for binaries?

Indeed. We're getting there. ;)

zgornel · 2019-07-06T07:23:57Z

@hhaensel , indeed, it looks like a permission problem as I cannot reproduce the problem on a second machine.
So,
* `libz.so` is a symlink indeed, lilke `libz.so.1` (both point to the same file)

* the `ENV["BINARYPROVIDER_COPYDEREF"]=false` or copying the lib in place of the symlink do not work

* wiping out the `.julia` directory **solves the problem**

* I do not see any way of tracing this issue back to something specific so far
  It may be that the problem is not related to `BinaryProvider`. Thanks for the assistance.
It should have been ENV["BINARYPROVIDER_COPYDEREF"]=true ;-)
But if you found that replacing the symlinks by hardcopies did not help the error has probably been elsewhere...

My bad, the value was true (it did not use symlinks). Edited the comment for future ref.

hhaensel · 2019-07-06T20:34:27Z

@BoundaryValueProblems
I was a bit fast in writing "delete the deps folder". I meant delete everything inside the deps folder except the build.jl.
If this doesn't help or if you have already deleted the deps folder, please also delete the folder 9jDi1 inside CodecZlib and then run ]build -v JLD2 or ]build -v CodecZlib
If there are any errors, please report them.

hhaensel · 2019-07-06T20:42:53Z

@staticfloat
if isolate=true makes the difference, then it seems that the environments of the REPL and Base.julia_cmd() are different. And even stranger, according to @zgornel 's post above

dl_esc_path = "/home/zgornel/.julia/packages/CodecZlib/9jDi1/deps/usr/lib/libz.so"
 success(`$(Base.julia_cmd()) -e "import Libdl; Libdl.dlopen(\"$(dl_esc_path)\")"`)

returns true at the REPL, but not when run from the build process.
I am afraid that anything concerning the package manager is above my abilities and I am probably out of the game here.
If there is anything concerning the windows patch don't hesitate to involve me again ;-)

hhaensel · 2019-07-06T21:00:51Z

@staticfloat , maybe it would be good to establish some kind of version checking for binaries?

Indeed. We're getting there. ;)

O wow! That's indeed a big thing. Thanks for pointing to that.

Crown421 · 2019-07-06T21:16:14Z

I deleted everything in the deps folder, and ran build -v CodecZlib and received the same error (on WSL)

Building CodecZlib → `~/.julia/packages/CodecZlib/9jDi1/deps/build.log`
[ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-linux-gnu.tar.gz to /home/steffen/.julia/packages/CodecZlib/9jDi1/deps/usr/downloads/Zlib.v1.2.11.x86_64-linux-gnu.tar.gz...
[14:07:24] ######################################################################## 100.0%
ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/steffen/.julia/packages/CodecZlib/9jDi1/deps/usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
 [1] error(::String) at ./error.jl:33

Same when deleting the entire 9jDi1 folder.

I can't test this right now on the linux server that I use, as I need it to run.

I should also mention that I put a completely fresh install of julia and all packages on the server the other day, and ran into the described issue creating my baseline environment. No updates or actual use had happened yet.

hhaensel · 2019-07-06T23:18:18Z

Very strange!
Just did a fresh install of both julia-1.1.0 and julia-1.1.1 on WSL Ubuntu on an updated Windows 10 64bit and everything works fine.
Perhaps rename your .julia for a while and retry ]add CodecZlib and build -v CodecZlib.

(v1.1) pkg> build -v CodecZlib
  Building CodecZlib → `~/.julia/packages/CodecZlib/9jDi1/deps/build.log`

(v1.1) pkg> st
    Status `~/.julia/environments/v1.1/Project.toml`
  [944b1d66] CodecZlib v0.5.2

(v1.1) pkg> st -m
    Status `~/.julia/environments/v1.1/Manifest.toml`
  [b99e7846] BinaryProvider v0.5.5
  [944b1d66] CodecZlib v0.5.2
  [3bb67fe8] TranscodingStreams v0.9.4
  [2a0f44e3] Base64
  [8ba89e20] Distributed
  [b77e0a4c] InteractiveUtils
  [8f399da3] Libdl
  [56ddb016] Logging
  [d6f4376e] Markdown
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [8dfed614] Test

Crown421 · 2019-07-06T23:48:39Z

Once I am done with the current project I will give reinstalling julia on the server a try.
Given that that the facts (deleting .julia helps, and I ran into the problem from a fresh install), some other package may create the conflict if installed first.
Unfortunately I do not remember the order in which I installed the packages, but I will check the packages that I have and compare against your list.

hhaensel · 2019-07-08T09:39:57Z

That was a fresh installation. All packages listed in the manifest are installed by the ]add CodecZlib.
Good luck!

BoundaryValueProblems · 2019-07-10T02:30:06Z

@hhaensel : I tried by moving my ~/.julia to ~/.julia.bak and installed various packages. But when it tried to install CodecZlib, it failed with the same error. I only tried this on my Windows 10 machine. By the way, I didn't reinstall julia v1.1.1; I just used the previously installed julia v1.1.1. Anyways, I will stick with [email protected] for a while.

BoundaryValueProblems · 2019-08-22T05:59:32Z

Today, I installed Julia v1.2.0, and hence, I also did fresh install of various packages. BinaryProvider v0.5.6 was installed automatically, and it didn't complain anything. So, perhaps, now is a good time to close this issue?

hhaensel · 2019-08-22T06:46:50Z

You are right, we can close this issue.
The root cause is not yet solved, though. BinaryProvider v0.5.6 is just a rename of v0.5.4 in order to overcome the problems of the introduction of the isolate mode.
There are some improvements in v0.5.5, such as symlinks on windows, which currently don't work. But this is pursued in #172, so feel free to close this :-)

BoundaryValueProblems · 2019-08-22T16:56:03Z

OK, then let's close this issue since #172 is still open. But somehow I myself cannot close it. Could someone close this? Thanks!

BoundaryValueProblems mentioned this issue Jul 5, 2019

v0.5.2 build problem on Windows JuliaWeb/LibCURL.jl#72

Closed

hhaensel mentioned this issue Jul 8, 2019

v0.5.5 breaks ReadStat.jl on windows #172

Closed

thofma mentioned this issue Jul 9, 2019

Build error v0.14.1 Nemocas/Nemo.jl#600

Closed

hhaensel mentioned this issue Jul 19, 2019

leaving the quoting up to the success command #173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CodecZLib build fails on Linux and WSL #170

CodecZLib build fails on Linux and WSL #170

Crown421 commented Jul 4, 2019 •

edited

Loading

BoundaryValueProblems commented Jul 4, 2019 •

edited

Loading

zgornel commented Jul 5, 2019 •

edited

Loading

hhaensel commented Jul 5, 2019 •

edited

Loading

zgornel commented Jul 5, 2019

hhaensel commented Jul 5, 2019

hhaensel commented Jul 5, 2019

zgornel commented Jul 5, 2019

hhaensel commented Jul 5, 2019

hhaensel commented Jul 5, 2019 •

edited

Loading

hhaensel commented Jul 5, 2019

hhaensel commented Jul 5, 2019

hhaensel commented Jul 5, 2019 •

edited

Loading

zgornel commented Jul 5, 2019 •

edited

Loading

hhaensel commented Jul 5, 2019 •

edited

Loading

hhaensel commented Jul 5, 2019

BoundaryValueProblems commented Jul 5, 2019

hhaensel commented Jul 5, 2019

staticfloat commented Jul 6, 2019

zgornel commented Jul 6, 2019 •

edited

Loading

hhaensel commented Jul 6, 2019

hhaensel commented Jul 6, 2019

hhaensel commented Jul 6, 2019

Crown421 commented Jul 6, 2019

hhaensel commented Jul 6, 2019

Crown421 commented Jul 6, 2019

hhaensel commented Jul 8, 2019

BoundaryValueProblems commented Jul 10, 2019

BoundaryValueProblems commented Aug 22, 2019

hhaensel commented Aug 22, 2019

BoundaryValueProblems commented Aug 22, 2019 •

edited

Loading

CodecZLib build fails on Linux and WSL #170

CodecZLib build fails on Linux and WSL #170

Comments

Crown421 commented Jul 4, 2019 • edited Loading

BoundaryValueProblems commented Jul 4, 2019 • edited Loading

zgornel commented Jul 5, 2019 • edited Loading

hhaensel commented Jul 5, 2019 • edited Loading

zgornel commented Jul 5, 2019

hhaensel commented Jul 5, 2019

hhaensel commented Jul 5, 2019

zgornel commented Jul 5, 2019

hhaensel commented Jul 5, 2019

hhaensel commented Jul 5, 2019 • edited Loading

hhaensel commented Jul 5, 2019

hhaensel commented Jul 5, 2019

hhaensel commented Jul 5, 2019 • edited Loading

zgornel commented Jul 5, 2019 • edited Loading

hhaensel commented Jul 5, 2019 • edited Loading

hhaensel commented Jul 5, 2019

BoundaryValueProblems commented Jul 5, 2019

hhaensel commented Jul 5, 2019

staticfloat commented Jul 6, 2019

zgornel commented Jul 6, 2019 • edited Loading

hhaensel commented Jul 6, 2019

hhaensel commented Jul 6, 2019

hhaensel commented Jul 6, 2019

Crown421 commented Jul 6, 2019

hhaensel commented Jul 6, 2019

Crown421 commented Jul 6, 2019

hhaensel commented Jul 8, 2019

BoundaryValueProblems commented Jul 10, 2019

BoundaryValueProblems commented Aug 22, 2019

hhaensel commented Aug 22, 2019

BoundaryValueProblems commented Aug 22, 2019 • edited Loading

Crown421 commented Jul 4, 2019 •

edited

Loading

BoundaryValueProblems commented Jul 4, 2019 •

edited

Loading

zgornel commented Jul 5, 2019 •

edited

Loading

hhaensel commented Jul 5, 2019 •

edited

Loading

hhaensel commented Jul 5, 2019 •

edited

Loading

hhaensel commented Jul 5, 2019 •

edited

Loading

zgornel commented Jul 5, 2019 •

edited

Loading

hhaensel commented Jul 5, 2019 •

edited

Loading

zgornel commented Jul 6, 2019 •

edited

Loading

BoundaryValueProblems commented Aug 22, 2019 •

edited

Loading