fetchTree: handle tarballs with multiple top-level files

[fgaz]

Motivation

See #7083. While the issue can be worked around by using the file type, it imposes a dependency on nixpkgs and makes implementation of other fetchers based on downloadTarball more difficult.

Context

Issue #7083 suggests two solutions to this problem:

1. If there are multiple top-level files, use the archive root
2. A stripRoot attribute, like in nixpkgs' fetchzip

This patch implements (1). It is forward-compatible with (2) if a missing stripRoot is interpreted as "automatic".

Fixes #7083

Priorities

Add 👍 to pull requests you find important.

[roberth]

Needs a test in tests/tarball.sh

[edolstra]

I'd prefer not to do this. The lazy-trees branch completely changes how tarballs are handled (they're imported into a local git cache and accessed from there), and it's not clear how to handle tarballs with more than one root object.

[fgaz]

@edolstra Why isn't it clear? Is it because we don't know whether to strip the root until all entries are processed? If so, what about a stripRoot parameter?

[thufschmitt]

what about a stripRoot parameter?

@fgaz That seems to be a good solution (defaulting it to true and mentioning it in the error message if there's more than one toplevel dir)

[roberth]

and it's not clear how to handle tarballs with more than one root object.

libgit should make this quite easy. Assuming you want to have a streaming implementation, you stream all the contents into blobs, and keep all directory entries, in the original tarball layout at first (probably in memory, because maintaining such a data structure on disk would be inefficient - could be done though, don't care). At the end, you check whether the raw root contains a single directory. If it does, you return its contents instead of the root.
Note that in both tarball and zip you have to read at least the archive end to determine the raw root contents, so streaming is only about minimizing memory usage, not speeding up access.

what about a stripRoot parameter?

I'm not so sure about that. Less "configuration" is generally better.

[fricklerhandwerk]

Triaged in Nix team meeting:

• @edolstra: could merge this part of lazy trees and collaborate with @fgaz on getting that feature into it
  • can do it even simpler, and always create a directory, and if that ends up only containing a directory, throw away the excess layer
• @roberth: getting more of that WIP merged would generally be great
• decision: do it as discussed

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-24-nix-team-meeting-minutes-106/35955/1

[mrtnvgr]

+1

[fgaz]

I just noticed #9485 contains the lazy trees part that was mentioned by @fricklerhandwerk. @edolstra do you accept PRs targeting your branch? Should I wait until #9485 lands?

[pinpox]

Are there any workarounds on how to handle zip flake inputs with mutiple files in the meantime?

[fgaz]

@pinpox You could fetch as file and import from derivation.

---

#9485 was merged, so I guess this is unblocked?

can do it even simpler, and always create a directory, and if that ends up only containing a directory, throw away the excess layer

[pinpox]

@pinpox You could fetch as file and import from derivation.

Hm, true, but has the downside of having to provide and update the hash myself instead of it being managed in flake.lock. I think this is still very relevant and an important feature.

[thufschmitt]

has the downside of having to provide and update the hash myself instead of it being managed in flake.lock

You can still have it as a flake input using the file input type