Run pytest tests in parallel for dm-haiku, optax, tensorflow-datasets, flax

[samuela]

Description of changes

My other 127 cores are lonely 😢

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.05 Release Notes (or backporting 21.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[samuela]

Result of nixpkgs-review pr 166163 run on x86_64-linux 1

2 packages failed to build:

• python39Packages.elegy
• python39Packages.treex

4 packages built:

• python39Packages.dm-haiku
• python39Packages.flax
• python39Packages.optax
• python39Packages.tensorflow-datasets

[samuela]

elegy and treex are pre-existing failures but will be fixed by #166167

[FRidh]

It should use $NIX_BUILD_CORES instead of auto.

We may also add a setup hook to pytest-xdist that sets this, as long as it is possible to disable it again as well.

[samuela]

It should use $NIX_BUILD_CORES instead of auto.

Ok, fixed in the latest series of commits!

[SuperSandro2000]

We may also add a setup hook to pytest-xdist that sets this, as long as it is possible to disable it again as well.

Yeah, really. This is often done wrong or forgotten.

[samuela]

Yeah, really. This is often done wrong or forgotten.

This sounds like a good idea. A quick search turned up 41 files that contain "pytest-xdist" but not "pytestFlagsArray":

• pkgs/development/python-modules/extractcode/default.nix
• pkgs/development/python-modules/scipy/default.nix
• pkgs/development/python-modules/ansible-runner/default.nix
• pkgs/development/python-modules/typer/default.nix
• pkgs/applications/audio/quodlibet/default.nix
• pkgs/development/python-modules/pytest-django/default.nix
• pkgs/development/tools/misc/gef/default.nix
• pkgs/top-level/python2-packages.nix
• pkgs/development/python-modules/django-oauth-toolkit/default.nix
• pkgs/development/python-modules/pip-tools/default.nix
• pkgs/development/python-modules/nipype/default.nix
• pkgs/development/python-modules/ocrmypdf/default.nix
• pkgs/development/python-modules/mitmproxy/default.nix
• pkgs/top-level/python-packages.nix
• pkgs/development/python-modules/commoncode/default.nix
• pkgs/development/python-modules/lazy_import/default.nix
• pkgs/development/python-modules/apache-beam/default.nix
• pkgs/development/python-modules/wandb/default.nix
• pkgs/development/python2-modules/hypothesis/default.nix
• pkgs/development/python-modules/aria2p/default.nix
• pkgs/development/python-modules/requests/default.nix
• pkgs/tools/admin/awscli2/default.nix
• pkgs/development/python2-modules/pytest-xdist/default.nix
• pkgs/applications/science/misc/cwltool/default.nix
• pkgs/applications/graphics/dosage/default.nix
• pkgs/development/compilers/vyper/default.nix
• pkgs/development/tools/analysis/checkov/default.nix
• pkgs/development/python-modules/pyppeteer/default.nix
• pkgs/development/python-modules/typecode/default.nix
• pkgs/development/python-modules/dyn/default.nix
• pkgs/top-level/python-aliases.nix
• pkgs/development/python-modules/pikepdf/default.nix
• pkgs/tools/security/cve-bin-tool/default.nix
• pkgs/development/python-modules/pytest-order/default.nix
• pkgs/development/python-modules/keras/default.nix
• pkgs/development/python-modules/psautohint/default.nix
• pkgs/development/python-modules/pytest-json-report/default.nix
• pkgs/development/python-modules/plugincode/default.nix
• pkgs/tools/admin/lexicon/default.nix
• pkgs/development/python-modules/pytest-xdist/default.nix
• pkgs/development/python-modules/librouteros/default.nix

[samuela]

I'll go ahead and merge tomorrow unless anyone objects

[SuperSandro2000]

This sounds like a good idea. A quick search turned up 41 files that contain "pytest-xdist" but not "pytestFlagsArray":

We get a setupHook for it so we don't need to track those down anymore.

[samuela]

We get a setupHook for it so we don't need to track those down anymore.

Meaning we don't need to specify pytestFlagsArray manually anymore?

[SuperSandro2000]

Yes, #166486