Windows CI using AppVeyor - also change to a MinGW binary #135

tkelman · 2014-08-25T20:27:31Z

How to get started:
Sign up at appveyor.com for a free open source plan
Probably have to authenticate to let AppVeyor see your GitHub repos, I forget
Click New Project, find this repo in the list and click Add
Push a test commit, or manually go to projects, HDF5.jl, click New Build

Also recommended so Travis and AppVeyor coexist nicely on pull request status indicators:
Follow instructions here http://github-multi-status.herokuapp.com/ to add an extra webhook for Status events

coveralls · 2014-08-25T20:35:15Z

Coverage remained the same when pulling 509b3be on tkelman:appveyor into 83f4152 on timholy:master.

coveralls · 2014-08-25T20:39:23Z

Coverage remained the same when pulling 509b3be on tkelman:appveyor into 83f4152 on timholy:master.

tkelman · 2014-08-25T22:04:04Z

I'm combining multiple somewhat unrelated commits into one PR here - the second commit switches the Windows binary to a MinGW one. I downloaded the binary from here http://sourceforge.net/projects/msys2/files/REPOS/MINGW/x86_64/mingw-w64-x86_64-hdf5-1.8.13-1-any.pkg.tar.xz but the libhdf5 dll also depended on libszip-0.dll which was in a separate download here http://sourceforge.net/projects/msys2/files/REPOS/MINGW/x86_64/mingw-w64-x86_64-szip-2.1-1-any.pkg.tar.xz (and similar for 32 bit, just replacing x86_64 with i686)

I tried to add both to the bindeps script, and use the unpacked_dir option I added to the Binaries provider a while back to simplify the custom unpacking. Unfortunately both downloaded tar.xz files extracted into the same folder and they didn't want to overwrite one another so I couldn't get both to extract properly. I had to repackage a single download that contained both (and while I was at it I put both 32 bit and 64 bit in different folders of the same file, just for simplicity).

coveralls · 2014-08-25T22:10:53Z

Coverage remained the same when pulling f635ba1 on tkelman:appveyor into 83f4152 on timholy:master.

tkelman · 2014-08-25T22:24:46Z

I see you were able to turn on AppVeyor. This passed tests locally for me with a Win64 Julia, but on AppVeyor apparently there were connectivity troubles (probably transient) downloading the Julia binary. And I should've checked the dependencies of the Win32 binaries more carefully (didn't test those locally, haven't downloaded and installed a Win32 copy of 0.3.0 yet). The 32-bit MinGW download I grabbed was compiled with the wrong kind of MinGW - one using dwarf2 exception handling, which is also a different C runtime than used by Julia (SJLJ on Win32). I'll keep searching.

simonster · 2014-08-25T22:36:20Z

Also note that the reason the mmap tests currently pass on Windows is probably that we now write small datasets in compact form. This is slightly faster (actually in terms of IO it is much faster, but for such small datasets the time to read/write is dominated by libhdf5) and consumes slightly less space, but compact datasets can't be mmapped because libhdf5 won't tell us where they are in the file. I should add a test that writes enough data that it will actually be mmapped (EDIT: done in 0b19961), but then we'll probably have to disable it on Windows for now.

coveralls · 2014-08-25T22:41:05Z

Coverage remained the same when pulling 17bb81c on tkelman:appveyor into 83f4152 on timholy:master.

tkelman · 2014-08-25T22:42:43Z

@simonster was wondering why that would start passing.

Do you have admin access here, are you able to turn on the multi-status webhook? It looks much better than having Travis and AppVeyor overwrite each other.

simonster · 2014-08-25T22:45:52Z

I don't think I have admin access (I don't think it's possible to grant it for repositories under a personal account), so @timholy will have to do this.

tkelman · 2014-08-25T22:50:21Z

On 32-bit:

ERROR: box: argument is of incorrect size
while loading C:\projects\hdf5-jl\test\jld.jl, in expression starting on line 452
while loading C:\projects\hdf5-jl\test\runtests.jl, in expression starting on line 5

simonster · 2014-08-25T22:56:17Z

Should be fixed in d07833f

from arch cross-compiled repository using SJLJ on Win32 instead of dwarf2 (msys2 version was using dwarf2, didn't work)

tkelman · 2014-08-25T23:02:12Z

As expected the mmap test fails on Windows, should be made @unix_only

WARNING: Error reading bigdata
ERROR: could not create file mapping: Access is denied. 
 in error at error.jl:21
 in mmap_array at mmap.jl:144
 in readmmap at C:\Users\appveyor\.julia\v0.3\HDF5\src\plain.jl:1338
 in read_vals at C:\Users\appveyor\.julia\v0.3\HDF5\src\JLD.jl:363
 in read_array at C:\Users\appveyor\.julia\v0.3\HDF5\src\JLD.jl:353
 in read at C:\Users\appveyor\.julia\v0.3\HDF5\src\JLD.jl:301
 in read at C:\Users\appveyor\.julia\v0.3\HDF5\src\JLD.jl:284
 in anonymous at no file:171
 in include at boot.jl:245
 in include_from_node1 at loading.jl:128
 in include at boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at client.jl:285
 in _start at client.jl:354
while loading C:\projects\hdf5-jl\test\jld.jl, in expression starting on line 292
while loading C:\projects\hdf5-jl\test\runtests.jl, in expression starting on line 5

coveralls · 2014-08-25T23:03:15Z

Coverage remained the same when pulling ffb2e3b on tkelman:appveyor into 3233ca5 on timholy:master.

simonster · 2014-08-25T23:05:30Z

And disabled.

tkelman · 2014-08-25T23:36:57Z

And green builds on Win32 https://ci.appveyor.com/project/timholy/hdf5-jl/build/1.0.7/job/3br9a236jgnhc47p
and Win64 https://ci.appveyor.com/project/timholy/hdf5-jl/build/1.0.7/job/osrcljk8ilbqhqut

Windows CI using AppVeyor - also change to a MinGW binary

simonster · 2014-08-25T23:43:34Z

Awesome! Thanks for taking care of this!

tkelman · 2014-08-25T23:51:43Z

HTH. Hope @ihnorton doesn't mind us switching out the old MSVC binary to a different one. Maybe we should get rid of the readme's suggestion to manually install the MSVC version of the hdf5 dll now that we've seen it causes crashes?

Also reminder to @timholy to turn on the multi-status webhook when he gets a chance, and may as well add Travis and AppVeyor and PkgEval status badges to the readme for better visibility. I think only the owner of the AppVeyor project can actually get the badge url from the settings page.

timholy · 2014-08-26T00:52:28Z

are you able to turn on the multi-status webhook

Happy to do this. I found the page to edit the webhooks, but I don't see anything labeled "multi-status". There's one labeled "status" and another "deployment_status". Am I on the right track?

And, @tkelman, awesome work! Indeed I'll edit the README.

simonster · 2014-08-26T00:53:05Z

I just edited the README. (Thanks for pointing that out!) Is there any easy way to check if Julia and libhdf5 are built against the same runtime? I'm betting the answer is no, but if it's possible we could make a runtime mismatch just slow things down instead of resulting in segfaults.

timholy · 2014-08-26T00:54:09Z

Just noticed you tackled that already, thanks @simonster!

tkelman · 2014-08-26T00:58:21Z

@timholy sorry I need to write clearer instructions - go to http://github-multi-status.herokuapp.com, click the link where it says "To get started: follow this link", authorize the app to see statuses, then copy the URL. Go to GitHub settings, add a new webhook with the copied Payload URL, "Leave the Content type and Secret as-is, then select Let me select individual events. and select the Status checkbox. Make sure the Active checkbox is ticked, and click on Add webhook."

@simonster sometimes it's hard to tell, especially if the runtime gets linked statically. Usually I check the dll via Dependency Walker to see whether it's linking to one of the msvcr###.dll's, or libgcc_s_*.dll, etc. Since we don't really have a universally present equivalent to ldd or otool on Windows.

timholy · 2014-08-26T01:02:23Z

@tkelman, should be done.

tkelman · 2014-08-26T01:05:48Z

Cool, we'll see when the next PR gets opened. The AppVeyor badge url should be available under settings at https://ci.appveyor.com/project/timholy/hdf5-jl/settings/badges

timholy · 2014-08-29T20:17:29Z

I'm ready to tag a new version. If people have previously installed HDF5 with the MSVC version, will this work for them? I.e., will it download & use the new MingGW version? I assume this will work since the library has changed names, but I am just being paranoid and checking.

tkelman · 2014-08-29T21:10:02Z

Good question. I'm trying it out now, will let you know.

tkelman · 2014-08-29T21:15:31Z

Okay, looks like this will probably happen after users update:

julia> Pkg.checkout("HDF5")
INFO: Checking out HDF5 master...
INFO: Pulling HDF5 latest master...
INFO: No packages to install, update or remove

julia> Pkg.test("HDF5")
INFO: Computing test dependencies for HDF5...
INFO: No packages to install, update or remove
INFO: Testing HDF5
ERROR: Library not found. See the README for installation instructions.
 in error at error.jl:21
while loading C:\Users\Tony\.julia\HDF5\src\plain.jl, in expression starting on line 42
while loading C:\Users\Tony\.julia\HDF5\src\HDF5.jl, in expression starting on line 1
while loading C:\Users\Tony\.julia\HDF5\test\runtests.jl, in expression starting on line 1

Maybe improving that message to suggest trying Pkg.build("HDF5") would help?

timholy · 2014-08-29T22:06:45Z

Won't the build script run automatically when the package updates?

tkelman · 2014-08-29T22:12:18Z

I think so, but I'm not completely sure.

timholy · 2014-08-29T22:12:57Z

Let's try it and see! 😄

tkelman · 2014-08-29T22:19:58Z

Yeah, looks good.

julia> Pkg.update()
INFO: Updating METADATA...
INFO: Updating cache of HDF5...
INFO: Computing changes...
INFO: Upgrading HDF5: v0.3.0 => v0.4.0
INFO: Building HDF5
INFO: Attempting to Create directory C:\Users\Tony\.julia\HDF5\deps\downloads
INFO: Directory C:\Users\Tony\.julia\HDF5\deps\downloads already created
INFO: Downloading file https://sourceforge.net/projects/mingw-w64-archlinux/files/x86_64/m
ingw-w64-hdf5-1.8.13-1-any.pkg.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   389    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0   420    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0   407    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0   491    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
  5 2840k    5  159k    0     0  12062      0  0:04:01  0:00:13  0:03:48 38116

tkelman force-pushed the appveyor branch from c5ddb77 to 509b3be Compare August 25, 2014 20:28

tkelman mentioned this pull request Aug 25, 2014

WIP: Write Julia types as HDF5 compound types #132

Merged

12 tasks

tkelman changed the title ~~Windows CI using AppVeyor~~ Windows CI using AppVeyor - also change to a MinGW binary Aug 25, 2014

tkelman added 2 commits August 25, 2014 15:57

Windows CI using AppVeyor

6bdf30b

switch to a mingw version of libhdf5 binary

ffb2e3b

from arch cross-compiled repository using SJLJ on Win32 instead of dwarf2 (msys2 version was using dwarf2, didn't work)

tkelman force-pushed the appveyor branch from 17bb81c to ffb2e3b Compare August 25, 2014 22:59

simonster added a commit that referenced this pull request Aug 25, 2014

Merge pull request #135 from tkelman/appveyor

a743a7c

Windows CI using AppVeyor - also change to a MinGW binary

simonster merged commit a743a7c into JuliaIO:master Aug 25, 2014

tkelman deleted the appveyor branch August 25, 2014 23:45

tkelman mentioned this pull request Nov 30, 2014

HDF5 Package Version 0.46 Can't install at Windows Julia0.3 or Julia0.4‏ #183

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows CI using AppVeyor - also change to a MinGW binary #135

Windows CI using AppVeyor - also change to a MinGW binary #135

tkelman commented Aug 25, 2014

coveralls commented Aug 25, 2014

coveralls commented Aug 25, 2014

tkelman commented Aug 25, 2014

coveralls commented Aug 25, 2014

tkelman commented Aug 25, 2014

simonster commented Aug 25, 2014

coveralls commented Aug 25, 2014

tkelman commented Aug 25, 2014

simonster commented Aug 25, 2014

tkelman commented Aug 25, 2014

simonster commented Aug 25, 2014

tkelman commented Aug 25, 2014

coveralls commented Aug 25, 2014

simonster commented Aug 25, 2014

tkelman commented Aug 25, 2014

simonster commented Aug 25, 2014

tkelman commented Aug 25, 2014

timholy commented Aug 26, 2014

simonster commented Aug 26, 2014

timholy commented Aug 26, 2014

tkelman commented Aug 26, 2014

timholy commented Aug 26, 2014

tkelman commented Aug 26, 2014

timholy commented Aug 29, 2014

tkelman commented Aug 29, 2014

tkelman commented Aug 29, 2014

timholy commented Aug 29, 2014

tkelman commented Aug 29, 2014

timholy commented Aug 29, 2014

tkelman commented Aug 29, 2014

Windows CI using AppVeyor - also change to a MinGW binary #135

Windows CI using AppVeyor - also change to a MinGW binary #135

Conversation

tkelman commented Aug 25, 2014

coveralls commented Aug 25, 2014

coveralls commented Aug 25, 2014

tkelman commented Aug 25, 2014

coveralls commented Aug 25, 2014

tkelman commented Aug 25, 2014

simonster commented Aug 25, 2014

coveralls commented Aug 25, 2014

tkelman commented Aug 25, 2014

simonster commented Aug 25, 2014

tkelman commented Aug 25, 2014

simonster commented Aug 25, 2014

tkelman commented Aug 25, 2014

coveralls commented Aug 25, 2014

simonster commented Aug 25, 2014

tkelman commented Aug 25, 2014

simonster commented Aug 25, 2014

tkelman commented Aug 25, 2014

timholy commented Aug 26, 2014

simonster commented Aug 26, 2014

timholy commented Aug 26, 2014

tkelman commented Aug 26, 2014

timholy commented Aug 26, 2014

tkelman commented Aug 26, 2014

timholy commented Aug 29, 2014

tkelman commented Aug 29, 2014

tkelman commented Aug 29, 2014

timholy commented Aug 29, 2014

tkelman commented Aug 29, 2014

timholy commented Aug 29, 2014

tkelman commented Aug 29, 2014