Hh/no errors #166

hhaensel · 2019-06-15T08:26:37Z

Suppress errors and warnings and correct quotes for Windows

…tar error message for windows 10 during `probe_platform_engines!()`

staticfloat · 2019-06-15T15:20:48Z

src/Products.jl

@@ -165,7 +165,8 @@ function locate(lp::LibraryProduct; verbose::Bool = false,
                if platforms_match(platform, platform_key_abi())
                    if isolate
                        # Isolated dlopen is a lot slower, but safer
-                        if success(`$(Base.julia_cmd()) -e "import Libdl; Libdl.dlopen(\"$dl_path\")"`)
+                        dl_esc_path = replace(dl_path, "\\"=>"\\\\")


Ooh, we gotta be careful with this one; I don't think this will work on every platform. What is more likely to work is to replace(dl_path, "\\"=>"/"), since forward-slash (even though it's not the default returned by tools on Windows) should work everywhere.

I would not expect that any other platform than windows will have a\ in the path.
Moreover, I just copied the idea of line 395

Just checked whether your proposal would work on windows, and interestingly it does pass the test. But ImageMagick does not build with "\\"=>"/" whereas it does with "\\" => "\\\\".
So I would encourage someone with a Linux machine to also test whether ImageMagick builds successfully with my version.

staticfloat · 2019-06-16T01:25:54Z

What is the image magick error?

…

On Sat, Jun 15, 2019 at 15:15 hhaensel ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/Products.jl <#166 (comment)> : > @@ -165,7 +165,8 @@ function locate(lp::LibraryProduct; verbose::Bool = false, if platforms_match(platform, platform_key_abi()) if isolate # Isolated dlopen is a lot slower, but safer - if success(`$(Base.julia_cmd()) -e "import Libdl; Libdl.dlopen(\"$dl_path\")"`) + dl_esc_path = replace(dl_path, "\\"=>"\\\\") Just checked whether your proposal would work on windows, and interestingly it does pass the test. But ImageMagick does not build with "\\"=>"/" whereas it does with "\\" => "\\\\". So I would encourage someone with a Linux machine to also test whether ImageMagick builds successfully with my version. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#166?email_source=notifications&email_token=AAA762F4IBLIEG3J63WCP23P2VSXLA5CNFSM4HYOROZ2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB3U6BQY#discussion_r294060932>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA762GINMUU7GB7Q2M3BBDP2VSXLANCNFSM4HYOROZQ> .

hhaensel · 2019-06-16T10:15:59Z

 Error: Error building `ImageMagick`:
│ WARNING: replacing module Anon.
│ ERROR: LoadError: LoadError: LibraryProduct(nothing, ["libpng16"], :libpng, "Prefix(C:\\Users\\helmu\\.julia\\packages\\ImageMagick\\AdMiw\\deps\\usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at .\error.jl:33
│  [2] #write_deps_file#169(::Bool, ::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at C:\Users\helmu\.julia\dev\BinaryProvider\src\Products.jl:415
│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at .\none:0
│  [4] top-level scope at none:0
│  [5] include at .\boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at .\loading.jl:1041
│  [7] include at .\sysimg.jl:29 [inlined]
│  [8] include(::String) at C:\Users\helmu\.julia\packages\ImageMagick\AdMiw\deps\build.jl:32
│  [9] top-level scope at C:\Users\helmu\.julia\packages\ImageMagick\AdMiw\deps\build.jl:33
│  [10] include at .\boot.jl:317 [inlined]
│  [11] include_relative(::Module, ::String) at .\loading.jl:1041
│  [12] include(::Module, ::String) at .\sysimg.jl:29
│  [13] include(::String) at .\client.jl:388
│  [14] top-level scope at none:0
│ in expression starting at C:\Users\helmu\.julia\packages\ImageMagick\AdMiw\deps\build_libpng.v1.6.31.jl:48
│ in expression starting at C:\Users\helmu\.julia\packages\ImageMagick\AdMiw\deps\build.jl:25
└ @ Pkg.Operations C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Pkg\src\Operations.jl:1069

Strangely, the test BinaryProvider passes. When I don't do any replacement, it fails with

ERROR: LoadError: LibraryProduct(nothing, ["libfoo"], :libfoo, "Prefix(C:\\Users\\helmu\\.julia\\dev\\BinaryProvider\\test\\LibFoo.jl\\deps\\usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] #write_deps_file#169(::Bool, ::Bool, ::Function, ::String, ::Array{Product,1}) at C:\Users\helmu\.julia\dev\BinaryProvider\src\Products.jl:415
 [3] write_deps_file(::String, ::Array{Product,1}) at C:\Users\helmu\.julia\dev\BinaryProvider\src\Products.jl:396
 [4] top-level scope at C:\Users\helmu\.julia\dev\BinaryProvider\test\LibFoo.jl\deps\build.jl:48
 [5] include at .\boot.jl:317 [inlined]
 [6] include_relative(::Module, ::String) at .\loading.jl:1041
 [7] include(::Module, ::String) at .\sysimg.jl:29
 [8] exec_options(::Base.JLOptions) at .\client.jl:229
 [9] _start() at .\client.jl:421
in expression starting at C:\Users\helmu\.julia\dev\BinaryProvider\test\LibFoo.jl\deps\build.jl:34
LibFoo.jl: Error During Test at C:\Users\helmu\.julia\dev\BinaryProvider\test\runtests.jl:978
  Got exception outside of a @test
  failed process: Process(`'C:\Julia\Julia-1.0.1\bin\julia.exe' -Cnative '-JC:\Julia\Julia-1.0.1\lib\julia\sys.dll' --compile=yes --depwarn=yes --code-coverage=none --color=yes deps/build.jl`, ProcessExited(1)) [1]
  Stacktrace:
   [1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at .\error.jl:42
   [2] pipeline_error at .\process.jl:705 [inlined]
   [3] #run#504(::Bool, ::Function, ::Cmd) at .\process.jl:663
   [4] run at .\process.jl:661 [inlined]
   [5] (::getfield(Main, Symbol("##79#83")){String})() at C:\Users\helmu\.julia\dev\BinaryProvider\test\runtests.jl:986
   [6] cd(::getfield(Main, Symbol("##79#83")){String}, ::String) at .\file.jl:85
   [7] macro expansion at C:\Users\helmu\.julia\dev\BinaryProvider\test\runtests.jl:980 [inlined]
   [8] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Test\src\Test.jl:1083 [inlined]
   [9] top-level scope at C:\Users\helmu\.julia\dev\BinaryProvider\test\runtests.jl:979
   [10] include at .\boot.jl:317 [inlined]
   [11] include_relative(::Module, ::String) at .\loading.jl:1041
   [12] include(::Module, ::String) at .\sysimg.jl:29
   [13] include(::String) at .\client.jl:388
   [14] top-level scope at none:0
   [15] eval(::Module, ::Any) at .\boot.jl:319
   [16] macro expansion at .\logging.jl:317 [inlined]
   [17] exec_options(::Base.JLOptions) at .\client.jl:219
   [18] _start() at .\client.jl:421

hhaensel · 2019-06-17T09:34:09Z

Some more hints: With respect to the critical call success($(Base.julia_cmd()) -e "import Libdl; Libdl.dlopen("$(dl_esc_path)")") I tested different calls of dlopen at the command prompt:

.julia\packages\ImageMagick\AdMiw\deps\usr>julia -e "import Libdl; Libdl.dlopen(\"bin/libpng16.dll\")"
ERROR: could not load library "bin/libpng16.dll"
The specified module could not be found.

Stacktrace:
 [1] dlopen(::String, ::UInt32) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdl
ib\v1.0\Libdl\src\Libdl.jl:97 (repeats 2 times)
 [2] top-level scope at none:0

.julia\packages\ImageMagick\AdMiw\deps\usr>julia -e "import Libdl; Libdl.dlopen(\"bin\\libpng16.dll\")"

.julia\packages\ImageMagick\AdMiw\deps\usr>

This seems to show that "/" is not accepted by dlopen() if it is passed with -e at the command line on windows systems. It works fine at the REPL though!

julia> Libdl.dlopen("bin/libpng16.dll")
Ptr{Nothing} @0x00000000693c0000

hhaensel · 2019-06-17T13:44:24Z

I traced part of the problem:
Different dll's seem to behave differently. The loading of the libfoo.dll from BinaryProvider properly loads with "/" in the path, whereas libpng16.dll from ImageMagick doesn't ... ?
With"\\" both dlls can be loaded.

julia> import Libdl

julia> cd("C:\\Users\\hh\\.julia\\dev\\BinaryProvider\\test\\LibFoo.jl\\deps\\usr")

julia> Libdl.dlopen("bin/libfoo.dll")
Ptr{Nothing} @0x0000000065100000

julia> cd("C:\\Users\\hh\\.julia\\packages\\ImageMagick\\AdMiw\\deps\\usr\\")

julia> Libdl.dlopen("bin/libpng16.dll")
ERROR: could not load library "bin/libpng16.dll"
The specified module could not be found.
"
Stacktrace:
 [1] dlopen(::String, ::UInt32) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Libdl\src\Libdl.jl:97 (repeats 2
times)
 [2] top-level scope at none:0

julia> Libdl.dlopen("bin\\libpng16.dll")
Ptr{Nothing} @0x00000000693c0000

julia> Libdl.dlopen("bin/libpng16.dll")
Ptr{Nothing} @0x00000000693c0000

hhaensel · 2019-06-17T15:19:35Z

I finally digged it down.
libpng16.dll tries to load libz-1.2.11.dll and for this looks in the same directory as libpng16.dll and in the current directory. So if I copy libz-1.2.11.dll to the working directory, then Libdl.dlopen("bin\\libpng16.dll") succeeds.
So I assume that libpng16.dll looks up the path of the caller and appends \libz-1.2.11.dll. This will fail, because of the mixed usage of \ and /.
@SimonDanisch might comment on this

All in all, I think the choice of "\\" => "\\\\" is not so bad ...

Another point, I came across is

BinaryProvider.jl/test/LibFoo.jl/deps/build.jl

Line 34 in 6b34650

if any(!satisfied(p; verbose=verbose) for p in products)

Maybe, it would be good to use isolate=true in this call, as satisfied() has isolate=false as default?
Otherwise, one would first check satisfied() in the current context, but during write_deps_file() require isolate=true and this might fail...
I found this confusing.

hhaensel and others added 2 commits June 15, 2019 10:20

Escape path and use different quotes for windows OS

dbf3265

Suppress symlink warning during probe_symlink_creation(...); avoid …

ad1ce20

…tar error message for windows 10 during `probe_platform_engines!()`

staticfloat reviewed Jun 15, 2019

View reviewed changes

hhaensel mentioned this pull request Jun 17, 2019

Symlinks on windows, again. JuliaIO/LibpngBuilder#8

Open

staticfloat merged commit 7168f68 into JuliaPackaging:master Jun 18, 2019

This was referenced Jun 18, 2019

Symlink warning on Windows 10 #149

Closed

Symbolic links on Windows #123

Closed

hhaensel deleted the hh/no_errors branch June 19, 2019 00:39

hhaensel mentioned this pull request Jul 5, 2019

CodecZLib build fails on Linux and WSL #170

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hh/no errors #166

Hh/no errors #166

hhaensel commented Jun 15, 2019

staticfloat Jun 15, 2019

hhaensel Jun 15, 2019

hhaensel Jun 15, 2019

staticfloat commented Jun 16, 2019 via email

hhaensel commented Jun 16, 2019

hhaensel commented Jun 17, 2019

hhaensel commented Jun 17, 2019

hhaensel commented Jun 17, 2019

Hh/no errors #166

Hh/no errors #166

Conversation

hhaensel commented Jun 15, 2019

staticfloat Jun 15, 2019

Choose a reason for hiding this comment

hhaensel Jun 15, 2019

Choose a reason for hiding this comment

hhaensel Jun 15, 2019

Choose a reason for hiding this comment

staticfloat commented Jun 16, 2019 via email

hhaensel commented Jun 16, 2019

hhaensel commented Jun 17, 2019

hhaensel commented Jun 17, 2019

hhaensel commented Jun 17, 2019