local evaluation: use nixpkgs ci.eval parallel implementation

[GaetanLepage]

This PR aims at improving local evaluation.

Currently, it uses a (single-threaded) nix-env call per system to evaluate.
This new patch leverages the recent ci.eval parallel evaluation implementation in nixpkgs itself.

Current limitations:

• Not respecting --allow as we used to (maybe it is possible...)
• It requires carefully tuning the --max-jobs, --cores and chunkSize parameters of ci.full
  We need to expose new options for those and provide sensible (automatic ?) default values.
• The listing of "Impacted packages" on stdout is now limited to listing all "rebuilds" (i.e. added + changed packages). Before, it was more detailed (removed, changed and added I think). This information is not available in the output of the eval.compare derivation. If needed, we could change the upstream implem...

---

TODO

• Handle --allow user preferences
• Fix tests which all got broken by this patch
• (Automatically) set sensible default values for the eval.full parameters (--max-jobs, --cores and --arg chunkSize)

cc @Mic92 @zowoq @khaneliman

[GaetanLepage]

For now, I initialize the parameters following the recommendations of the ci.eval README:

• --max-jobs -> n_systems
• --cores -> n_cpu // n_systems (n_cpu is the number of logical cores, i.e. threads on the system)
• chunkSize -> 10_000

I conducted early and naive benchmarks on the following system:

• 6 cores / 12 threads (Ryzen 5600X)
• 64GB of RAM

I ran the evaluation for the four supported systems (x86_64-linux, aarch64-linux, x86_64-darwin, aarch64-darwin):

method	duration (mm:ss)	max RAM usage	number of cores fully used
legacy nix-env	2:25	53 GB	4 (1 per system evaluated)
new ci.eval.full	4:50	39GB	12

So unfortunately, it appears that this method is slower than our current one... Maybe tuning the hyper-parameters can give better results, but as we are maxing out the CPU, I don't expect a lot of gains.
Nevertheless, the new method allows for finer control of RAM usage (not everyone has 64GB of RAM).

Note: using chunkSize=20000, the time went to 4:25.

cc @infinisil, maybe you will have more insights.

[infinisil]

You should be able to get the same speed as the legacy nix-env by setting the chunk size to like 1000000 (there's about 100000 attributes right now) at least, but to get the best speed I don't have a better answer than just to try a bunch of different numbers for the chunk size

[Mic92]

How evenly is the max memory usage spread across different chunks?
After all I consider this also a parameter that is useful just for keeping memory usage within certain boundaries.

[GaetanLepage]

How evenly is the max memory usage spread across different chunks?

I don't really know. The overall RAM usage fluctuates and peaks very briefly at ~39GB (using chunkSize = 10k).
I haven't made more precise measures.

[Mic92]

It feels like we need to solve this in nix. i.e. track RAM usage there and restart the interpreter after a threshold is breached.

[GaetanLepage]

So, what should we do about this ?
While I find this new implem more satisfying, cleaner and allowing finer control over the evaluation process, one must admit that it is significantly less performant.

[Mic92]

So, what should we do about this ? While I find this new implem more satisfying, cleaner and allowing finer control over the evaluation process, one must admit that it is significantly less performant.

Maybe the new method could be a different branch for the time being. Some people might have machines with enough resources, where it actually speeds up things. I am wondering if we can leverage libnix somehow to restart evaluation when it consumes to much RAM.

[infinisil]

While I find this new implem more satisfying, cleaner and allowing finer control over the evaluation process, one must admit that it is significantly less performant.

Did you try my suggestion of having a single huge chunk, so that it should effectively be the same single-threaded performance as before?

[GaetanLepage]

Did you try my suggestion of having a single huge chunk, so that it should effectively be the same single-threaded performance as before?

Sorry, I forgot to follow up.

Here is the benchmark I made:

• Current implementation (1 nix-env process per system, i.e. 4 total):

________________________________________________________
Executed in  362.20 secs    fish           external
   usr time   18.08 mins    0.27 millis   18.08 mins
   sys time    1.73 mins    1.04 millis    1.73 mins

• ci.eval.full-based implementation
  • chunkSize: 200k
  • --max-jobs: 4 (as we are evaluation on 4 systems)
  • --cores: 3 (my system has 12 = 3*4 cores). This should make no difference considering the chunkSize value, only a single process will be spawned for each platform.

Well...it OOMed :'(
I killed it after

________________________________________________________
Executed in  323.28 secs    fish           external
   usr time    3.58 secs    0.00 micros    3.58 secs
   sys time    6.26 secs  843.00 micros    6.26 secs

And it was not done with the first (pre-change) evaluation...

[infinisil]

I can only imagine that nixpkgs-review doesn't do the same eval then