211-character store path limit

[dhess]

Describe the bug

I'm running into Nix's 211-character store path limit when packaging a simple React app:

error: store path 'w0p04i8klrmla412sxk0539dpx45c408-qpl7imy79n9sd9a2h6s7armj9vqaal70-rk74xqfkqgc282lxhym3vwd9ww35iv7n-9sp1q8q5p3r6h2qm4g74ycp1f7b44zv7-9ikdajp8imc685n5vmiib3f9lny2ygz0-0xnc8c7z7rzxp1hf0vfdjwhhvwd19q3b-helper-builder-binary-assignment-operator-visitor-7.14.5.tgz' has a name longer than 211 characters

To Reproduce

Unfortunately the source code for my app is not public, but everything has been going fine until I tried to include Tailwind via CRACO. You may be able to reproduce the same problem I'm having by using npmlock2nix to package the example app described here:

https://tailwindcss.com/docs/guides/create-react-app

Specifically, it's when I include the Tailwind dependencies that the problem arises:

https://tailwindcss.com/docs/guides/create-react-app#install-tailwind-via-npm

Expected behavior

I've filed an issue on Nix asking why this limit exists:

NixOS/nix#5086

Barring a Nix fix, is there any kind of workaround I can use at the npmlock2nix level?

Environment

• OS name + version: macOS Big Sur
• Version of the code: 3820a83

[andir]

cc @zimbatm I think this is the issue I mentioned the other day but neither of us could reproduce it.

[andir]

@dhess I think I've once seen this with the fetchers and we should probably supply (or shorten if we already do) the name attribute of all the sources we fetch. IIRC this is only an issue with builtin fetchers and not those that come from nixpkgs. Would that make sense in your scenario?

[dhess]

@dhess I think I've once seen this with the fetchers and we should probably supply (or shorten if we already do) the name attribute of all the sources we fetch. IIRC this is only an issue with builtin fetchers and not those that come from nixpkgs. Would that make sense in your scenario?

Yes, that sounds right.

Given that you've possibly seen this issue before and you're not able to reproduce it, and given that there's nothing at all secret about this app at this stage, I'll make a copy of it availably publicly in the hopes that you can reproduce it.

[andir]

@dhess I think I've once seen this with the fetchers and we should probably supply (or shorten if we already do) the name attribute of all the sources we fetch. IIRC this is only an issue with builtin fetchers and not those that come from nixpkgs. Would that make sense in your scenario?

Yes, that sounds right.

Given that you've possibly seen this issue before and you're not able to reproduce it, and given that there's nothing at all secret about this app at this stage, I'll make a copy of it availably publicly in the hopes that you can reproduce it.

Thanks! It would be enough to publish a simple example app as you linked to above. I'd want to add it as regression test to the repo then (with the fix).

[dhess]

Here it is:

https://github.com/hackworthltd/react-nix-craco-public

It's quite basic — just the default TypeScript app created by create-react-app, plus some basic dev dependencies. Everything was cool until the last commit which adds Tailwind via CRACO, and that's where we run into the above issue.

edit Sorry, I should have mentioned that the problem occurs when you, e.g., nix-build -A packages.x86_64-linux.frontend

[zimbatm]

@andir what we tested was on a builtin derivation. Maybe builtins.fetchurl has a different rule.

[dhess]

FYI, Nix seems unlikely to extend this 211-character limit because ext4's filename limit is only 255 characters (see NixOS/nix@693e68e), so I think this'll need to be handled in npmlock2nix.

[andir]

FYI, Nix seems unlikely to extend this 211-character limit because ext4's filename limit is only 255 characters (see NixOS/nix@693e68e), so I think this'll need to be handled in npmlock2nix.

Yeah, I'll add a fix & test to #82 tonight.

[andir]

@dhess I looked at your case again. After throwing away your lockfile I wasn't able to reproduce that exact error anymore. Instead I am seeing another issue with NPM not picking up the correct URL from the lockfile. I have no idea where that comes from.

Let me know if you are available for a bit of a chat / interactive session in one of the coming days. I tried to message you on Matrix (@dhess:matrix.org) from my account (@andi:kack.it).

[dhess]

In chat, @andir pointed out that my package-lock.json has Nix store paths in it, which is odd, because I didn't do that on purpose. Neither of us knows how this happened, but it first occurred in this commit:

https://github.com/hackworthltd/react-nix-craco-public/commit/b0e5b32f2fe55b6780dd6c29813c85e82f4dde97

@andir thinks that I'm hitting the 211-character limit because of these Nix store paths in the lockfile.

I was able to recreate the package-lock.json file so that it contains no Nix store paths in this commit:

https://github.com/hackworthltd/react-nix-craco-public/commit/37b852373a507fe5b7419c1ce2c92cbae14f5593

I'm now working on adding Tailwind again with the new lockfile.

[dhess]

In chat, @andir pointed out that my package-lock.json has Nix store paths in it, which is odd, because I didn't do that on purpose. Neither of us knows how this happened, but it first occurred in this commit:

hackworthltd/react-nix-craco-public@b0e5b32

This happens when you're using node_modules_mode = "copy" and node_modules exists when you npm install --save-dev --package-lock-only in the project's Nix shell. I verified this behavior and fixed it in https://github.com/hackworthltd/react-nix-craco-public/commit/4f92ea72c7410f667461a325f637924352794a47

[infinisil]

You can replace https://github.com/tweag/npmlock2nix/blob/5fc6505adec1bafc3ae9ebd2a28673f6f5ed9a75/internal.nix#L27-L30 with lib.strings.sanitizeDerivationName, which then also takes care of ensuring it doesn't exceed the store path length:

https://github.com/NixOS/nixpkgs/blob/36b02f89feb239d7c0e52c9f661beb8cf69d0ebd/lib/strings.nix#L746

[infinisil]

Does this still occur after #150?

[dhess]

I’m not using this builder at the moment, so I can’t say.