python3Packages.galois: init at 0.0.32

[ChrisPattison]

Description of changes

• Add galois https://github.com/mhostetter/galois
• Add maintainer chrispattison

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[ChrisPattison]

I am currently working on upstreaming the numpy version constraint patch
mhostetter/galois#398

[ChrisPattison]

The numpy version constraint is due to the current numba release not supporting numpy 1.23 numba/numba#8263

Nixpkgs patches numba per #181620 (comment)

[mweinelt]

That looks pretty good already. Only a few marks on what I think can still be improved.

[ChrisPattison]

All changes implemented. Thanks for the feedback!

[mweinelt]

How long do the tests run for you? I added pytest-xdist to checkInputs and it was just under 4 minutes on an older 32C/64T machine. If tests are run single-threaded I expect quite some test runtime.

[ChrisPattison]

They take about 10 minutes on my Icelake macbook. As far as I am aware, they are single threaded. Would you recommend pytest-xdist? I guess it runs the tests in parallel?

[mweinelt]

Adding pytest-xdist parallelizes pytest tests runs, indeed. Our package automatically provides a hook that sets up pytest to use xdist with $NIX_BUILD_CORES. I don't believe galois contains anything racy, like file access, so xdist should be fine. Yes, I recommend it in this case.

[ChrisPattison]

I was unable to get pytest to parallelize the tests. Using nix-shell --pure --cores 4 and setting enableParallelBuilding = true for good measure (should this be set anyway?) the tests seemed to be run in serial despite xdist being listed as a plugin for pytest:

============================= test session starts ==============================
platform darwin -- Python 3.9.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /private/tmp/nix-build-python3.9-galois-0.0.32.drv-0/source, configfile: setup.cfg, testpaths: tests
plugins: xdist-2.5.0, forked-1.4.0
collected 7323 items

Any thoughts? As an aside, this doesn't seem to be documented anywhere

[mweinelt]

First you see the hook getting executed and then a number of gateways started that are used for tests.

python3.10-galois> pytestXdistHook
python3.10-galois> pytestCheckPhase
python3.10-galois> Executing pytestCheckPhase
python3.10-galois> ============================= test session starts ==============================
python3.10-galois> platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
python3.10-galois> rootdir: /build/source, configfile: setup.cfg, testpaths: tests
python3.10-galois> plugins: xdist-2.5.0, forked-1.4.0
python3.10-galois> gw0 [7323] / gw1 [7323] / gw2 [7323] / gw3 [7323] / gw4 [7323] / gw5 [7323]
[...]
python3.10-galois> ======================= 7323 passed in 461.91s (0:07:41) =======================

As an aside, this doesn't seem to be documented anywhere

Yeah, oops. Should be part of docs/language-frameworks/python.section.md, but it's missing.

[mweinelt]

I was unable to get pytest to parallelize the tests.

What is nix.buildCores set to? This is what the hook uses to determine parallel test runners.

Using nix-shell --pure --cores 4 and setting enableParallelBuilding = true for good measure (should this be set anyway?)

Doesn't really matter unless you're actually compiling anything. The tests don't benefit from that setting.

[ChrisPattison]

I have the tests running in parallel on my nixos box, so it must be the nix configuration as you mentioned. It takes about 5 minutes on a 4C/8T Zen 3 machine with about half that time stuck on 1-2 long running tests.

Overall, I think it is good to merge now?

[mweinelt]

Yes, it was. I just wanted to make sure everything was working for you as expected.