Conversation
|
The last time I looked into this, it was almost possible to process the lock files directly in Nix and download each dependency directly in a granular fashion rather than having to bundle it all in one large fixed‐output derivation. denoland/deno#19512 was the issue I ran into; it’s since been closed as fixed, but I remember when I looked at the fix I wasn’t confident it was enough to fully close up the issues. Do you think you’d be up for looking into whether that approach might work now? I think it’d be a much nicer solution if it turns out to be possible. I can try to publish the incomplete draft Nix code I came up with while playing with that if it’d be of any use. |
| fetchFromGitHub, | ||
| ... | ||
| }: | ||
| buildDenoApplication rec { |
There was a problem hiding this comment.
I think it would be ideal to support the buildDenoApplication (finalAttrs: { pattern. It's not the expected behavior yet but nice to have. Here is an example of a conversion of buildGoModule: https://github.com/NixOS/nixpkgs/pull/321791/files
| prePatch | ||
| patches | ||
| postPatch | ||
| ; |
There was a problem hiding this comment.
It might be possible to reduce this noise by using builtins.intersectAttrs (builtins.functionArgs fetchDenoDeps) args somewhere futher down.
| # pkgs/build-support/go/module.nix | ||
| # | ||
| # Looked at and didnt understand: | ||
| # pkgs/development/compilers/nim/build-nim-package.nix |
There was a problem hiding this comment.
build-nim-package.nix is convoluted and hard to understand because it uses fixed-point style right off, it does lockfile processing, and it has a function for overriding the lockfile.
The solution might be too much of a hack to be a good example but think the Nim problem is conceptually similar to Dino in that the Nim package description format and lockfiles are not reproducible by Nix standards. The current Nim solution is to use a tool to create a Nix specific lockfile when manually updating packages and commit the lockfile into Nixpkgs. There isn't anything so show that this will scale well into the future.
The solution that I've been working on is to reuse an independent software-bill-of-materials standard and add some Nix annotations but nothing has been merged to Nixpkgs. This comes with an alternative implementation of build-nim-package.nix - build-nim-sbom.nix.
|
@emilazy Thanks for the pointer to that issue. As far as I can tell, the redirect problem has been fixed with deno.lock v3, which records redirects. The second issue was about response headers, but I am not sure if they matter if we are fetching and building the dependencies independently from deno and are then supplying them as vendored dependencies. Deno recently gained a feature for vendoring dependencies, which is what I am currently using for building a fixed output derivation. However, I don't think it stores the received headers anywhere, so I don't think they matter when running with The bigger problem is that Deno sends its version number in the user-agent header of its http requests, and some sites change their response depending on the version. I put an example of this in the PR description. I would love to have a look at your draft of the Nix code; it is much appreciated. |
|
Yeah, I handled the user agent thing like this: downloadPath = fetchurl {
inherit url sha256;
# "By default, esm.sh checks the User-Agent header to determine the build target." - https://esm.sh/#docs
curlOptsList = ["--user-agent" "Deno/${deno.version}"];
};I don’t even have my old draft in version control right now, but I’ll try to get it very minimally cleaned up and published within the next few days; feel free to ping me here again if I don’t report back soon. It’s over a year old now and definitely not ready for prime‐time, but if Deno has evolved enough that we can actually use a pure Nix approach with fine‐grained dependencies like that it’d be a wonderful improvement on the huge fixed‐output derivation blobs approach most ecosystems are stuck with. |
|
FWIW the headers stuff mattered at least when interpreting URLs as JavaScript vs. JSON, and possibly for pointers to TypeScript type definitions; I had a proof‐of‐concept demonstrating the divergence, but I never got around to sharing it with the Deno people I don’t think. Maybe they changed behaviour enough by now that it no longer applies. It also might be marginal enough that we don’t have to care. |
|
@emilazy How is your progress with cleaning up your draft code? I would also be happy about a not cleaned up version. |
|
Here you go. I haven’t really done anything to update it but you can get the basic gist, at least. I think if you messed with the Note that while I think the developer experience of a non‐IFD, non‐FOD based solution like this is second to none, and it’d be a great synergy between the Deno and Nix ecosystems if it could be made to work elegantly, there is some backlash to vendoring lock files in Nixpkgs currently, so it may be that a FOD‐based approach like yours is more palatable for Nixpkgs use. Ideally, both would be supported, and share as much interface as possible. I hope the code is useful or at least interesting to you. Please let me know if you have any questions or want me to test something out. |
|
I mean this should be in a hook like the others (pnpm,yarn), I am planning to make a hook for deno but I think this should not have to be needed for a builder as hook basicly give you the overrides of what the hook does and it's just generally much better than a builder. |
|
Yes there could be a builder that uses the hook but is there any real need for a builder? |
Description of changes
This PR adds a proof of concept for a
buildDenoApplicationfunction similar to thebuildRustCratefunction. This is still work in progress.I went through the Deno manual, issues, and code and tried to collect all the information relevant to importing things and fetching dependencies based on the lockfile:
Imports
In ECMAScript modules, files are imported via import specifiers. They can be either relative (
./file.ts), bare (idk/file.ts), or absolute (prot:/absolute/path/file.ts). Deno supports importing via relative and absolute specifiers. Relative specifiers are evaluated relative to the current file. Deno supports thefile,data,blob,node,http,https,npm, andjsrprotocols for absolute specifiers.Deno also supports import maps, which enable us to define bare specifiers. Import maps are specified when running a script, usually via
deno.json, which means that libraries cannot use import maps.Deno can use lockfiles to lock the versions and verify the integrity of all external imports. Similarly to import maps, the lockfile is specified when running a script, so libraries cannot bring their own lockfiles.
Relative imports and imports from
file:specifiersLocal imports are unproblematic because they are not dependencies but our project's source. Local imports outside our src repo are out of scope for now.
Import from
node:specifiersDeno can import Node.js built-in modules. I am unsure how Deno does this, but it doesn't produce any files in the
$DENO_DIRor the vendored directories. The Node.js built-in modules are also built into Deno.Imports from
data:specifiersDo not need to be locked or included in dependencies.
Imports from
blob:specifiersBlob imports do not need to be locked or included in dependencies.
Imports from
https:(andhttp:) specifiersImports from URLs. The
deno.lockversion 3 solves the issue with redirects and directly lists all URLs we need to fetch. The more significant issue is caching only the fetched file contents but not the received HTTP headers. This can be a problem because Deno seems to change its behavior based on the headers, as documented in denoland/deno#19512. We can mitigate these issues by just ignoring all headers. I am not even sure if this is an issue anymore because I could not find any trace of the headers in the output ofdeno cache --vendor --node-modules-dir, so when we dodeno --cached-only runafterward, the behavior should be unaffected by the headers. Not sure if this leads to failing applications because the behavior changes when using--cached-only, but that would be a general deno problem, not a nix problem.Imports from
npm:specifiersDeno can import packages from npm registries. npm imports consist of a package name, a version requirement, and an optional path in the package.
If the version requirement of a package is a specific version, we can assume that this package is deterministic. I think this is an even stronger assumption than deterministic content behind URLs, as all relevant public registries have the policy that the content of a specific version is immutable ( (npm.js, deno.land/x, jsr.io). However, the dependencies of a version are usually based on relatively broad version requirements, which means that they need to be locked.
The Deno lockfile v3 sufficiently locks the packages. However, it seems like Deno only supports one npm registry at a time. The lockfile tracks the hash for every required package/version combination and the exact package/version combination of every resolved dependency. This is different from npm
package-lock.jsonfiles, which track incomplete versions for dependencies if multiple packages request different versions of the package. Deno does all the heavy lifting for us here; we don't need to determine which version a package uses.Deno supports using a vendored
node_modulesdirectory in the project directory. We need to create that directory with Nix as a part of the dependencies. All npm package sources live in directories likenode_modules/.deno/PACKAGE_NAME@VERSION/node_modules/MAYBE_PACKAGE_SCOPE/PACKAGE_NAME. There is nonode_modulesfolder in that folder. I am surprised that this works, but it does. See the list below for more details on the exact structure. There also are relative symlinks atnode_modules/MAYBE_PACKAGE_SCOPE/PACKAGE_NAMEfor every direct dependency of our packages that point to the correct version of the package innode_modules/.deno. I think the structure is relatively sane, as we have a clean directory without dependencies for every package. Everything else is just symlinks.MANGLED_PACKAGE_NAMErefers to a package name in which all slashes have been replaced with+. For example,@webview/webviewbecomes@webview+webview. As+is not allowed in normal package names, this can not produce collisions.PACKAGE_NAMErefers to a package's name. If it is a scoped name, it does not include the scope; for@scope/name, this is justname.MAYBE_PACKAGE_SCOPErefers to the scope of a package. If it is not a scoped package, this is just empty. For example, for@scope/name, this is@scope. For an unscoped package liketsx, this is nothing ``.The dependencies are stored and symlinked in the node_modules folder as follows:
node_modules/.deno/MANGLED_PACKAGE_NAME@VERSION/node_modules/MAYBE_PACKAGE_SCOPE/PACKAGE_NAMEis the directory that contains the actual npm package. It only includes the files fetched from npm and nonode_modulesfolder.node_modules/.deno/MANGLED_PACKAGE_NAME@VERSION/node_modules/MAYBE_DEPENDENCY_SCOPE/DEPENDENCY_NAMEis a symlink tonode_modules/.deno/MANGLED_DEPENDENCY_NAME@RESOLVED_VERSION/node_modules/MAYBE_DEPENDENCY_SCOPE/DEPENDENCY_NAME. These links exist for all direct dependency of the package innode_modules/.deno/MANGLED_PACKAGE_NAME@VERSION/node_modules/MAYBE_PACKAGE_SCOPE/PACKAGE_NAME. I am surprised that this structure works with Node.js imports, but it does.node_modules/MAYBE_DEPENDENCY_SCOPE/DEPENDENCY_NAMEis a symlink tonode_modules/.deno/MANGLED_DEPENDENCY_NAME@RESOLVED_VERSION/node_modules/MAYBE_DEPENDENCY_SCOPE/DEPENDENCY_NAME. It exists for every direct dependency of our Deno project.However, there are more files in
node_modules.node_modules/.deno/.setup-cache.binis a binary file that keeps track of the symlink structure innode_modules. Having this file isn't strictly necessary, as Deno will recreate it at runtime. However, Deno will fail if it cannot create that file at runtime, which it will if we load node_modules from the Nix store. In Deno 1.44, the file is also not deterministic because the entries are ordered randomly. I submitted a patch; this will be fixed in Deno 1.45.node_modules/.deno/.deno.lockandnode_modules/.deno/.deno.lock.pollare probably used as a mutex. They are deterministic. Deno does not mind if they are missing. Deno also does not mind if it can not write these. We don't need to include them in ournode_modulesfolder.node_modules/.bin/EXECUTABLE_NAMEare symlinks to all entries in thebinsections of all direct and indirect dependencies. I am still determining how Deno handles conflicts if multiple packages export executables with the same name, but it seems deterministic.node_modules/.deno/PACKAGE_NAME@VERSION/.initializedis an empty file for every npm package. I assume Deno uses these to track whether a package folder is ready to be used. These files do not strictly need to exist, as Deno will create them at runtime. However, Deno will fail if it can not create them because the directory is in the Nix store.I think these two lists are exhaustive, but I have not checked Deno's source code to verify that.
A bit tricky with using vendored node_modules is the
.setup-cache.binin the vendored folder, which maps module names to module names with versions, to keep track of the symlink structure in node_modules. It currently is a binary file without a clearly defined structure. I submitted a patch to deno to make it at least deterministic.Imports from
jsr:specifiersDeno can resolve
jsr:specifiers to import packages from jsr.io. Similar tonpm:imports,jsr:imports consist of a package name, a version requirement, and an optional path in the package.Deno will then decide the actual version for each jsr module and store them in the lockfile. Unlike npm packages, the concept of packages is only used to resolve the versions of dependencies. Afterward, Deno maps JSR imports to https URLs. If we know the resolved version, we can essentially replace a jsr import with the correct https import. So, for example,
jsr:@std/path/joinis equivalent tohttps://jsr.io/@std/path/0.225.2/join.ts. Note that it also added the.tsextension to the URL. JSR package version metadate can contain an export map that maps paths in the specifier to files in the package. In this case, the export map contains an entry like"./join": "./join.ts".JSR imports are basically a layer of syntactical sugar on top of
https:imports, with added support for version requirements. It also brings a few QoL improvements like the ability to use import maps to library authors, at the cost of only being able to publish at JSR because you need a build step (for various reasons).When vendoring
jsr:imports, the downloaded files are stored in thevendordirectory likehttps:imports. In fact, every file is placed in the correct location, corresponding to its URL. However, in addition to the TypeScript files, the vendor directory also containsvendor/jsr.io/SCOPE/PACKAGE/meta.jsonfor metadata about the available versions andvendor/jsr.io/SCOPE/PACKAGE/VERSION_meta.jsonfor every version of the package that we use. The version metadata file also includes an import map for the package.Annoyingly, files imported via
jsr:specifiers are not represented like https imports in the lockfile but in a data structure similar to npm modules.packages.specifierstracks the specifiers for all directly imported jsr and npm packages and their resolved versions. Each specifier's versions and resolved dependencies are tracked inpackages.npmandpackages.jsr, respectively. Both are objects withpackagename@resolvedversionas keys. The values are objects with two fields;integritycontains a checksum for the package, anddependenciescontains an array of specifiers of its dependencies. I have not yet found documentation on calculating the integrity checksum of JSR packages. Unlikenpmmodules, the specifiers in dependencies are not necessarilyabsolute, so something likejsr:@std/assert@^0.226.0instead ofjsr:@std/assert@0.226.0is possible. We can resolve these via thepackages.specifiersmap, but that is an extra step.I found the following reproducibility issues with the current vendoring strategy for JSR modules:
vendor/meta.jsoncontains all published versions of the package and is constantly changing because of this. Which is not deterministic at all. A file like this should not exist in the vendor directory because the information is not relevant for vendoring; it is only for caching when using vendored dependencies and only when updating them. This file must exist when running with--cached-onlydeno. For npm imports, this type of information is always stored in$DENO_DIRinregistry.jsonfiles. jsr imports should handle this in a similar fashion.My preferred solution for problems 2 and 3 is to include all files in vendor with their checksums in
deno.lock. This way, we also don't have to bother with thepackages.jsrstructure, as that is only needed for version resolution, which Deno already did for us.With this behavior, Deno reintroduces a few of the problems Node.js imports have.
Ways of triggering imports
I tested various ways of triggering imports to see if they make a difference for the lockfile.
JSX import with comment
Deno offers a few weird ways of importing jsx runtimes
In the above,
/jsx-runtimeis appended to the path and is treated like a normal import fromhttps://esm.sh/preact/jsx-runtime. We can also use imports with an npm specifier, in which case nothing is appended.The jsx source can also be defined in
deno.json; in that case, it is implicitly imported from all jsx and tsx files. This means that it is only imported if we import at least the tsx file.{ "compilerOptions": { "jsx": "react-jsx", "jsxImportSource": "https://esm.sh/preact" } }If we don't define a jsx import in
deno.jsonor via @jsxImportSource comment, deno will import nothing and fail with aReact not definederror when using tsx files.Dynamic imports
Deno supports dynamic imports via the import function. When doing dynamic imports with string literals as URLs, deno will cache and lock the dependencies. When an expression is used as the URL, deno is (obviously) unable to lock and cache that dependency.
However, when we are running deno with
--cached-only(which we are)Caches dependency, does not require networking permissions at runtime, works at runtime
Caches dependency; does not require networking permissions at runtime; works at runtime
Does not cache dependency; requires networking permissions at runtime; fails at runtime because the module is not in cache.
Caches dependency; works at runtime
Caches dependency; works at runtime
I am not sure about the behavior of dynamic imports. The table below shows some results of different combinations in different environments.
Testing the behavior of different dynamic imports
Various other things
HTTP headers
We have previously seen that Deno sometimes treats files differently depending on the received http headers. However, some package registries also treat us differently depending on the headers we send.
If we set our user agent to
Deno/1.44.4, we get a different result:If we set our user agent to
Deno/1.13.0, we again get a different result withdenoinstead ofdenonext:This shouldn't be a problem if we send the same headers as Deno. But which version? It's probably best if we either decide on one and keep it or use the same version as Deno.
This should not be a big problem as there are not many sites that do this and even on esm.sh we get responses with
denonextfor all recent versions and that is unlikely to change. I think we should just pin it to1.45.0and add a switch in the future. If we would use the correct Deno version, we would also have to write the version number into the output, which would mean that the outputHash would change on every Deno update.package.json support
I will just ignore that for now and abort if a project contains a package.json and no deno config. Although it should be possible and not that hard.
Private npm registries
We will need to add support for these, but not now.
https://docs.deno.com/runtime/manual/node/private_registries/
Proxies, private repos, and auth tokens
Deno supports HTTP and HTTPS proxies. I don't know how Nix handles proxies; further investigation is needed.
https://docs.deno.com/runtime/manual/basics/modules/proxies/
Deno provides a way to specify an auth token for specific domains. Further investigation is needed.
https://docs.deno.com/runtime/manual/advanced/private_repositories/
FFI / native code
Deno allows calling into native code using
Deno.dlopenbut does not provide a way of importing or packaging binaries. So, that is not part of the lockfile.Deno once had a plugin system, but that was removed in 1.13. Good, this way, we do not have to worry about that
WebAssembly
WebAssembly is usually fetched at runtime, so we don't need to worry about it. However, Deno may add it to the lockfile in the future, so we might need to keep an eye out for that.
TODO
Tasks in Deno
None of the these are strictly required and we should be able to work around all of them. However having those features in Deno would make things a lot cleaner and less hacky.
packages.jsrmeta.jsonfile like theregistry.jsonfiles and always store them $DENO_DIRnode_modulesdirectory are deterministicnode_modules/deno/.setup-cache.binto a JSON filevendor/manifest.json(feat: make the cache manifest more deterministic denoland/deno_cache_dir#53)meta.jsonfiles with--cached-only --lockregistry.jsonfiles with--cached-only --lockTasks in nixpkgs
buildRustCratebuildRustCrateClassic nixpkgs checklist
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.