Enhancement: migrate from nan to node-addon-api (N-API)

[Nantris]

I'm trying to use this with Electron and run some tests in Node but I absolutely cannot get a working version for each runtime working simultaneously. It always puts the built module into Release\sharp.node, overwriting the previously built version.

If I build for electron, I cannot test. If I build for Node, I cannot run it in Electron.

When I build for Electron, my tests return the following:

The file it says it cannot find does exist

 FAIL  app/redux/sagas/firebase/oneTime/__tests__/helpers.test.js
  ● Test suite failed to run

    The specified module could not be found.
    \\?\C:\projects\myApp\app\node_modules\sharp\build\Release\sharp.node

      Error: The specified module could not be found.
      \\?\C:\projects\myApp\app\node_modules\sharp\build\Release\sharp.node
      at Runtime.requireModule (node_modules/jest-runtime/build/index.js:372:31)
      at Object.<anonymous> (app/node_modules/sharp/lib/constructor.js:10:15)

When I build for Node, my app's renderer throws this error:

ELECTRON_ASAR.js:172 Uncaught Error: The system cannot find message text for message number 0x%1 in the message file for %2.
\\?\C:\projects\myApp\app\node_modules\sharp\build\Release\sharp.node
    at process.module.(anonymous function) [as dlopen] (ELECTRON_ASAR.js:172:20)
    at Object.Module._extensions..node (module.js:671:18)
    at Object.module.(anonymous function) [as .node] (ELECTRON_ASAR.js:172:20)
    at Module.load (module.js:561:32)
    at tryModuleLoad (module.js:504:12)
    at Function.Module._load (module.js:496:3)
    at Module.require (module.js:586:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (C:\projects\myApp\app\node_modules\sharp\lib\constructor.js:10:15)
    at Object.<anonymous> (C:\projects\myApp\app\node_modules\sharp\lib\constructor.js:259:3)

[lovell]

Hello, this is more a current limitation of native modules in Node rather than being a problem specific to sharp.

Different versions of Node and Electron each have different ABIs, hence the need to ensure compilation occurred against the relevant runtime.

Node provides a new ABI-stable N-API that aims to solve this problem. To take advantage of this we need to:

• Wait for Add support for N-API builds prebuild/prebuild#220 to land
• Wait for the N-API to be backported as "stable" to Node.js 8 to avoid warnings about its experimental behaviour
• Wait for Node.js 8 to reach EOL as N-API support is not quite stable enough in that major version.
• Migrate sharp from nan to node-addon-api

[Nantris]

Understood. Thanks very much for the explanation @lovell. Is there no way to use different release directories for different binaries under the current system? I get that there's no good way, but I'd be okay with hacking things together in the meanwhile. Unfortunately I wouldn't have the slightest clue how to do it, if it can be done at all.

It seems like some modules don't have this problem, gRPC for example. @murgatroid99, can you mention how gRPC approaches this? I know whatever gRPC and some other modules are doing works currently on Node >8.9.3, though perhaps there's just no differences in the binaries between Node and Electron in those cases?

[murgatroid99]

For gRPC we distribute separate binaries for Node and Electron, and node-pre-gyp has a feature that we use that parameterizes the name of the directory the binary is installed in based on which variant of the binary is being installed. So instead of creating build/Release/grpc_node.node, it creates something like src/node/v64-linux-x64-glibc/grpc_node.node or src/node/electron-2.0-windows-ia32-unknown/grpc_node.node.

[lovell]

sharp uses prebuild to create separate binaries for Node and Electron.

prebuild-install only downloads the relevant one at install-time and probably doesn't re-check at runtime, but it does have an install-time --runtime flag that might be worth experimenting with.

[Nantris]

Thanks very much to you both for your insightful answers.

[metabench]

@lovell Could anything be used as an alternative to prebuild?

"Migrate sharp from nan to node-addon-api" - That sounds like something that can happen independently of waiting for other things to be ready (depending on developers' interest in doing so).

Would there be a way to get Sharp to work on either node or electron without needing to be compiled differently?

[lovell]

The prebuild changes have landed (although it looks like the linked issue hasn't been updated to reflect that) so that's no longer a problem. N-API is still marked as experimental in Node 6 but the use of node-addon-api might alleviate that now. Happy to accept a PR if you're able.

[metabench]

I doubt I'll have time (or the right experience) to do this really soon. I've recently concluded that tensors provide a useful abstraction for image processing, and am going to focus on https://github.com/metabench/ta-tensor for the moment. I plan on making a C or C++ addon and also compiling to WASM for use in the browser. I'm interested in Sharp both because I've been using it, and to find out about best practises for making data-intensive addons. If it turns out that some new techniques I use would be useful to Sharp then I'd be in a better position to implement them in Sharp in the future.

[lovell]

Based on my slightly painful experience in another module of migrating to N-API across various versions of Node 8, where different versions of Node, Electron and N-API all report the same ABI but don't even support the same API, upgrading to use N-API will have to wait until after we remove support for Node 8 at the end of 2019.

[lovell]

An organisation that relies on sharp has very kindly offered to pay for the migration to N-API so expect to see some progress on this task over the next couple of weeks.

[lovell]

The latest work-in-progress for N-API migration is now available for early/alpha testing on the yield branch.

https://github.com/lovell/sharp/tree/yield

19 changed files with 567 additions and 711 deletions

npm install lovell/sharp#yield

[lovell]

v0.25.0 now available with the use of N-API. I'm sure the maintainers of the many modules that depend upon sharp will be happy with the reduced level of installation issues this will bring.