lib.linkedLists: init

[roberth]

Best way to return a large concatenated list from a foldl', and probably more use cases.

It's a means of last resort, but it's very helpful when needed.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[ConnorBaker]

Any thoughts on something like creating a balanced binary tree (rope) and a fold-tree implementation (like https://hal.science/hal-04580379/document)?

Alternatively, thoughts on using the Scott encoding of lists if you need O(1) eliminators? IIRC attribute sets are fairly expensive and there’s no special case for them like with small lists.

[roberth]

attribute sets are fairly expensive

I would prefer to make attrsets about equally inexpensive, which I feel more positive is achievable considering recent Nix contributor activity.

Similarly, I would like the lists in Nix to ultimately be more clever, because these aren't really things that should be the concern of Nix expressions.

• "What type of list should I use" should virtually never be a question if you're doing packaging and configuration.
• The interpreter isn't constrained by the purely functional programming model and could more easily implement other data structures, e.g. arrays up to a certain size and then a switch to whichever data structure proves best for our use cases.

Scott encoding of lists

That's worth a try, but also harder for contributors to understand as opposed to a cons that's expressed as a straightforward data type.
For the purpose of avoiding stack overflows on current Nix implementations I'm not sure whether Scott encoding helps or not. The reliance on function calls makes me suspicious, but I could be wrong, or maybe they could serve a purpose in part of the data structure or operations somehow. I haven't looked deeply into this.

[ConnorBaker]

As an aside, have you considered adding a tree-fold to lib (similar to the definitions for foldl and foldr)? For example,

foldt =
  op: nul: list:
  let
    # Fold over range [lo, lo+count)
    foldRange =
      lo: count:
      if count == 0 then
        nul
      else if count == 1 then
        elemAt list lo
      else
        # If the count is even, count/2 and (count+1)/2 sum back to count (due to integer division truncating the latter).
        # If the count is odd, count/2 and (count+1)/2 sum back to count (due to integer division truncating the former).
        # Computing them here avoids creating environment variables for them.
        op
          (foldRange lo (count / 2))
          (foldRange (lo + count / 2) ((count + 1) / 2));
  in
  foldRange 0 (length list);

With that, I was able to compute

length (foldt (a: b: a ++ b) [ ] (genList (n: genList (_: 0) n) 16000))

where lib.foldl and lib.foldr would consume lots of memory or have many, many GC cycles.

[infinisil]

Discussed with @roberth in a call, should have a practical application of this as motivation