haxe fails to build, depending on the path of libneko

[orivej]

Here we are building Haxe 3.4.4 with Neko 2.1.0. The build sometimes fails with:

(cd /build/haxe/extra/haxelib_src && /build/haxe/haxe client.hxml && nekotools boot run.n)
Uncaught exception - std@module_read
make: *** [Makefile:119: haxelib] Error 1

and when it fails, rebuilding Neko or Haxe does not help. Conversely when Haxe builds successfully, rebuilding Neko or Haxe does not make it fail. The minimal difference that fixes Haxe build is changing the path to Neko (by building and installing it with a different prefix). Here are two examples of paths to Neko that fail the build of Haxe:

/nix/store/4m2ik76hn1qimzqn81gp4badyjg2k10m-neko-2.1.0
/nix/store/s7gnh7vx0s23q719a4v06bv0lz8nb3y4-neko-2.1.0

The corresponding search path encoded into libneko.so in the latter case is:

/nix/store/s7gnh7vx0s23q719a4v06bv0lz8nb3y4-neko-2.1.0/lib/neko:/nix/store/s7gnh7vx0s23q719a4v06bv0lz8nb3y4-neko-2.1.0/bin

Here is an example of a path that supports the build of Haxe:

/nix/store/0ky4zn5a2cn26y99kz1ajf00ahbg7pc3-neko-2.1.0

[orivej]

This may also be affected by the install prefix of Haxe since it changes whenever the prefix of Neko changes, but that would be truly surprising.

[vcunat]

I've seen a couple packages doing clumsy pattern matching in CFLAGS and similar strings, being affected by some substrings of these hashes that get in there in nixpkgs.

[orivej]

Depending on the install prefix (probably), Neko itself fails to build with:

[ 89%] Generating bin/nekoml.std                                              
Uncaught exception - load.c(398) : Invalid module : /tmp/nix-build-neko-2.1.0.drv-0/neko-2.1.0-src/build/bin/nekoml.n
make[2]: *** [CMakeFiles/nekoml.std.dir/build.make:62: bin/nekoml.std] Error 1
make[1]: *** [CMakeFiles/Makefile2:1406: CMakeFiles/nekoml.std.dir/all] Error 2

(when nekoml.n is binary identical in succeeding and failing builds).

[andyli]

I don't have much idea about the build failure. I have just tried to reproduce that myself in a NixOS vm, but the build succeed.

For the neko build failure, Uncaught exception - load.c(398) : Invalid module, I got similar issue in the Fedora koji build system. I found that somehow the compiled .n files were broken, but they should be binary identical to the ones in boot/. I'm not sure if this is related to this issue, since you've mention that the failing nekoml.n output was the same as the succeeding one. But, you can still try to skip rebuilding the .n files and just use the ones in boot. You may apply HaxeFoundation/neko@39396e9 and HaxeFoundation/neko@ae0e560 as patches.

This may also be affected by the install prefix of Haxe since it changes whenever the prefix of Neko changes, but that would be truly surprising.

I think this is caused by the way haxelib is built. It is a produced by combining the neko executable with a neko bytecode file, thus it is not really valid linux binary: HaxeFoundation/neko#130
And the neko executable may contains the neko installation prefix (rpath?).
For haxe 3.4.4, haxelib can be compiled in another way:

1. make libs haxe to build the haxe compiler, without building haxelib
2. pushd extra/haxelib_src && cmake . && make && popd
cp extra/haxelib_src/haxelib haxelib

This will produce a haxelib that is linked with libneko, instead of embedding the whole neko executable. This is what homebrew is doing: https://github.com/Homebrew/homebrew-core/blob/master/Formula/haxe.rb#L43-L49

[orivej]

Thank you! Building haxelib in this way will likely solve the issue of building Haxe, and build failures of Neko are much less frequent. Please keep that NixOS VM, I'll write the reproduction instructions. (Although this is not restricted to NixOS, they will apply to Nixpkgs installed on other Linuxes as well.)

[orivej]

We have just updated neko package to 2.2, and unless you'd like to debug what was wrong before, we could wait a week or two to accumulate build history with the new version. (If a build failure happens, it will be reproducible because the previous were.) I hope that it solves the issue, so feel free to close it preemptively: I'll write here if it reoccurs; or keep it open and bump it later for updates.

[orivej]

One thing I'd like to know is: is it important that the default NEKO_PATH contains /usr/local/bin:/usr/bin? With neko 2.1 we were making it be $prefix/lib/neko:$prefix/bin, but with neko 2.2 we let NEKO_MODULE_PATH preset it with $prefix/lib/neko, so it no longer contains $prefix/bin.

[andyli]

It was used by nekotools boot ... to locate the neko binary, but since HaxeFoundation/neko@9f40f9e, nekotools also search in PATH, so it is not needed anymore. I've just tested in a NixOS vm and nekotools boot ... worked as expected.

[andyli]

Argh, accidentally closed the issue. Let me know if there is any remaining issue and I will reopen.

[orivej]

OK. A couple of rebuilds of haxe since our update of neko has all succeeded, but there is not enough of them to be certain, since such a short series of successes was also probable before the update.