Cross Compilation features

[lexor90]

Hello there,
and thank you for your awesome work.

Since we're planning to add compilation capabilities to our CI/CD environment we also have the needing to be able to cross-compile from various platforms to others.

I managed to add flags to specify valid node.js compilation platforms (both os and cpu arch):

./nodec --dest-os=linux --dest-arch=x64 index.js

I managed to get it to also cross-compile between mac and linux (with some limitations due to newer v8 versions, known things out there).

Valid architectures and CPUs are the same of the vanilla node.js (it should be checked with added dependencies, BTW).

In order to properly cross-compile one must use a valid toolchain, since this project always compiles from source it could also be provided inside the repo itself, what do you think about it?

Let me know if it looks good to you and your thoughts.

[pmq20]

Sorry for the delay. I'll look into it ASAP. Thanks for the PR.

[firrae]

With @pmq20 having v1 support C++ based native modules wouldn't this not work in most cases as native modules need to be built on their intended platforms? I'm pretty sure bcrypt compiled on Mac doesn't work on Linux, but maybe I'm wrong?

[lexor90]

@pmq20 Thanks, I'll rebase against the latest changes asap not sure about the windows failing compilation, but I think it might be already broken when I started working on the PR.

@firrae By using node.js from source code you basically get the same support you'd get for the node executable itself (node-compiler only wraps it by creating the temporary filesystem before to actually call the entry file). So you can refer to the official support, I guess, for possible issues with native modules. By the way you can find several docs online about node.js specific cross-compilation cases [1] which are pretty well covered.

In order to do so successfully of course you need to set up a cross compilation toolchain and properly configure it inside your shell environment.

The only issue I got from Mac was about the install process trying to execute some runtime scripts on the host machine – the infamous mkpeephole – because, of course, you can compile it and produce a valid binary for the destination platform, but it won't certainly run anything successfully on the host (compiling system) if it's different from the target platform. This is a know issue happening since v7 [2].

You can check code comments to see the options needed to avoid common cross compilation issues, I thought to enable them by default when cross compilation is detected

# --without-intl=none fixes: icutrim.py - it tries to run a binary made for linux on mac
# --cross-compiling, require host executables rather than target ones
# --without-snapshot avoids mksnapshot to run on host platform after build

I'm still stuck with the mkpeephole.

P.S. I also managed to get internally a way to define what node.js version to compile from, but since then the master has been updated with patches on the node code. So now it must be checked that those patches are compatible with v6.x before to be able to downgrade (at this point it might make sense to force the specific min and max versions supported by those patches).

@pmq20 If the patches are compatible with the v6.x line we could exploit git to patch the source code automatically (more or less the same way React Native does it [3]).

[1] https://github.com/netbeast/docs/wiki/Cross-Compile-NPM-modules
[2] nodejs/node#9707
[3] https://facebook.github.io/react-native/docs/upgrading.html

[firrae]

@lexor90 that's interesting, I've never had to look at cross-compiling before the project I'm currently on so thanks for the links! It's good to know.

[lexor90]

@firrae I just gave a quick look at the node-gyp cross-compilation status which is what native modules do use to compile themselves from sources and it seems there's still some issue with it [1] mainly because all of the libraries built upon it have no concept of cross-compiling, so they do assume that target_host === building_host [2]. I think there's more support for cross compilation across different processor architectures (eg. x86 to ARM) rather than across different operating systems though, thanks to the variety of embedded platforms and mobile devices that need to be targeted.

I know for sure there's some way to get bcrypt cross compiled properly (by exporting needed files from the target platform) but I never tried it though.

Maybe a quick workaround here would be to be able to run some pre defined script on the first run on your target machine, in order to install those modules and just link the libraries from the outside. Some sort of shared library.

But I don't actually know what kind of involvement would it mean on the implementation side and what kind of compatibility support would it retain with plain nodejs. I'd rather search for some solutions that may already be in use out there.

[1] nodejs/node-gyp#829
[2] nodejs/node-gyp#1137

[firrae]

Semi off topic:

Personally speaking, I'm in a bind where the idea of build once and deploy anywhere is almost a necessity so we've built out our own flow for it. The project I'm on has the need to be installed on isolated servers with no internet connection and limited build tools (usually it's a VM they spin up and tell us to install on) so having to build it on startup wouldn't help me, personally, but I bet there are plenty of other use cases where this would be greatly welcome. Once I solve my build issue using this (currently bcrypt doesn't seem to work for me) I'll likely set up a build pipeline that builds on each of our supported platforms.

Thanks for the links and info though @lexor90.

[firrae]

@lexor90 I noticed that you have a binary and a shell script in the bin directory of your PR, are those needed for something or just the results of a test that got included in a push?

[lexor90]

@firrae Yeah, you're right. I somehow missed I've checked them in.

I'm going to remove them as they are not related to the project itself, thanks.

BTW the bin/run.sh gives you an example of shell environment setup to cross-compile node.js (and more generically any C/C++ project). It basically rewrites the environment variables used by buildtools to use the executable we want rather than the system's default.

[lexor90]

@pmq20 can we setup a cross-compile toolchain on our CI in order to test this? I think the easiest cross-compilation would be changing the target processor (we do also support changing operating system, but it's quite tricky as the binary format does differ - for example mac uses mach-o, while linux uses elf).

If this looks good to you merge it, we're simply exposing the native support (we also need to point out to the official node bugtracker for any issue regarding cross-compilation as long as it's not concerned to one of our nodec-specific dependencies).

[pmq20]

@lexor90 May I share another possible approach on cross-compiling? @ngot has found out that simply attaching a chunk of data to the end of the executable does no harm to the executable itself. By utilizing this technique we could pre-compile a dozen of binaries for each arch and OS, and download them when compiling a user's project. The compiler could just compress the user's folder into a SquashFS image and attach it to the end of the downloaded binary and it is done. In this way, cross-compiling is easily achieved. Still need proof of concept but I think this direction is very promising.

[ngot]

here is the example :

• https://github.com/fibjs/fibjs/blob/master/fibjs/src/base/fibjs.cpp#L107
• https://github.com/fibjs/fibjs/blob/master/test/selfzip_test.js#L72

[lexor90]

Hey there,
that would be great. Seriously, it would get rid of what takes the most to compile.

I know that also pkg works with precompiled binaries (according to their readme) and I was at first looking at this possibility, as it would mean to have a much quicker compilation time, but then I could not find a way to make this happen since we need to compile nodec from source.

I'm surely going deeper into this approach, as it could be a nice solution also for #40.
Cross compilation would still be needed in order to compile properly the remaining part of nodec otherwise one would have a combination of nodejs and nodec compiled for two different platforms (this means you'd download nodejs for linux, but compile squashfs on mac os x which would obviously lead to a not working binary).

[lexor90]

@pmq20 if we are able to precompile npm alongside nodejs we could also get rid of the local npm command, so we don't need to check the current system-wide version of node. Am I right? The build would become then truly reproducible and working on any system with or without nodejs installed (at the moment we do depend on npm install before to actually compile node and libsquash). The only difference would be that npm must be run on the host platform (which is doing the compilation) while the precompiled node would match the target platform (which is where it must be run to).

[pmq20]

@lexor90 Yes. It should be easy since npm is written in Javascript instead of C++. My thinking is we divide the compile process into 2 passes. On the first pass a vanilla Node.js environment is compiled without any hack and produced and cached to be used later as a local node environment. On the 2nd pass, the user's project entrance is hacked into node and do the real compiling for the user.

[aemonge]

Such a wanted feature, is there any plans to merge this ?