RFC: Package Distributions #519
Conversation
| "node": "10" | ||
| }, | ||
| "platform": "win32", | ||
| "package": "[email protected]" |
There was a problem hiding this comment.
does this mean that "distributions" only points to additional packages? How will this impact dependency graph analysis tools?
| { | ||
| "package": "fs-readdir-native@1" | ||
| } |
There was a problem hiding this comment.
what condition applies here?
|
|
||
| ## Summary | ||
|
|
||
| Today, maintainers utilize various strategies to distribute platform-specific versions of their software under a singular package namespace. Often, these strategies rely on `install` scripts or `optionalDependencies` with some type of bootloader implementation (ex. [`esbuild`](https://npmjs.com/package/esbuild?activeTab=explore)); borth are not ideal. The `npm` CLI should support a first-class/standard way for maintainers to define conditions in which a package distribution is reified in place of the origin target. |
There was a problem hiding this comment.
Emphasis mine:
Often, these strategies rely on
installscripts oroptionalDependencieswith some type of bootloader implementation (ex.esbuild)
In the case of esbuild it should be and: It uses both optionalDependencies and a postinstall script:
- There is still a small post-install script but it's now optional in that the
esbuildpackage should still function correctly if post-install scripts are disabled (such as withnpm --ignore-scripts). This post-install script optimizes the installed package by replacing theesbuildJavaScript command shim with the actual binary executable at install time. This avoids the overhead of launching anothernodeprocess when using theesbuildcommand. So keep in mind that installing with--ignore-scriptswill result in a sloweresbuildcommand.
For a tool like esbuild – which can compile a small project in a couple of milliseconds – it’s a bit funny that the JavaScript wrapper around the executable can take more time than the compilation itself. If you only run esbuild once in a while it doesn’t matter super much of course.
But that extra startup time does add up in the Elm ecosystem. Editor plugins run both Elm and elm-format on save, and there you want an as snappy an experience as possible, so shaving 100 ms or so definitely counts.
Many people install elm-format with npm locally in each project (so that every contributor is on the exact same version, which is very important for formatters). So a JavaScript wrapper around the project hurts!
So – is this something that this RFC seeks to improve on too? (Just checking!) In my opinion it would be awesome to somehow achieve maximum performance and --ignore-scripts with no “shenanigans” to pull from package authors!
There was a problem hiding this comment.
Hmm … like this?
{
"name": "foo",
"version": "1.2.3",
"distributions": [
{
"platform": "win32",
"package": "[email protected]",
"bin": "./bin.js"
},
{
"platform": "linux",
"arch": "x64",
"package": "[email protected]",
"bin": "./foo-native-linux-x64"
},
"..."
]
}In other words – one "bin" per distributions object, pointing directly to the executables on Linux and macOS, and to a wrapper on Windows (Node.js based, or CMD based, or whatever you like, or maybe you can point directly to an .exe file even?)
If the above is correct, I’d love to see text about this added to the RFC!
Edit: The "bin" fields in foo points to files in foo-native-linux-x64 and other “sub” packages? 🤔
|
Hi there! Really happy to see this RFC and can provide an example from how we currently achieve the distribution of binaries at Prisma We support something we called [
'darwin',
'darwin-arm64',
'debian-openssl-1.0.x',
'debian-openssl-1.1.x',
'rhel-openssl-1.0.x',
'rhel-openssl-1.1.x',
'linux-arm64-openssl-1.1.x',
'linux-arm64-openssl-1.0.x',
'linux-arm-openssl-1.1.x',
'linux-arm-openssl-1.0.x',
'linux-musl',
'linux-nixos',
'windows',
'freebsd11',
'freebsd12',
'openbsd',
'netbsd',
'arm',
]
The download of the binaries is achieved on Here you can see there are quite a few requirements to replace this with
Happy to be part of a next meeting if you find this interesting 😃 |
|
I forgot to mention that we also need in some cases to install more than one "binaryTarget" locally. Example:
|
|
There are two features we need for Next.js that
Hopefully |
|
Besides detecting engines and other conditions automatically, also we need a mechanism to install dependencies for the other platform for cross-compiling: Refs:
|
On Linux, it is not possible to tell exactly what kind of C library a native modules depends on just by os/cpu, so yarn 3.2 and cnpm added libc fields to further distinguish this case. This avoids downloading both `gnu` and `musl` packages at the same time. Currently only [yarn 3.2+](yarnpkg/berry#3981) and [cnpm](cnpm/npminstall#387) are supported, the npm implementation is [still under discussion](npm/rfcs#519).
On Linux, it is not possible to tell exactly what kind of C library a native modules depends on just by os/cpu, so yarn 3.2 and cnpm added libc fields to further distinguish this case. This avoids downloading both `gnu` and `musl` packages at the same time. Currently only [yarn 3.2+](yarnpkg/berry#3981) and [cnpm](cnpm/npminstall#387) are supported, the npm implementation is [still under discussion](npm/rfcs#519).
| }, | ||
| { | ||
| "platform": "linux", | ||
| "arch": "x64", |
|
Feedback from users of next.js
Feedback from users of NAPI-RSNoway to install prebuild packages for another platform.This is really important for Electron developers and Docker users. For example, users need to build their Windows ia32 distribution Electron app on Windows x64 and they need to install prebuild packages for windows-ia32 platform on it. Yarn 3 is the only solution for this case at present: https://yarnpkg.com/configuration/yarnrc#supportedArchitectures. There are some issues mentioned about it:
Install wasm fallback without postinstall scriptAlways including the wasm fallback package as dependency will increase the installation size. There is no way to install the wasm fallback package without postinstall script when all the platform specified optionalDependencies are not suitable for the host platform. What we are doing for this case is to use postinstall script to detect if the native binaries are installed successfully, if not, we will install the wasm fallback package in the postinstall script.
|
|
@saquibkhan the use case I find valuable here is if I could make the "default" distribution for my package omit tests, source files, docs, etc, but provide some kind of "full" distribution that includes all those things. The beneficial install bandwith impact of applying this to my 300-400 packages alone would likely save npm millions of dollars a year, conservatively. |
Co-authored-by: Jordan Harband <[email protected]>
Co-authored-by: Steven <[email protected]>
Co-authored-by: Steven <[email protected]>
|
i've made some updates to this document including several unanswered questions as well as detailing a known risk and adding in some other alternatives |
Co-authored-by: Jordan Harband <[email protected]>
|
👋 @darcyclarke we can see that you closed this on Dec 1 but it's not clear why. npm/statusboard#647 says to refer to this PR for information. Is there somewhere that says why this RFC was closed? Thanks! |
|
@JoshuaKGoldberg I apologize for the wait but I was weighting a response against leaning into the exact situation I was trying to avoid (ie. being asked for updates on |
distributionscapabilitiesSee rendered RFC