Skip to content

Comments

Packaged extra JAX libs#152754

Closed
ndl wants to merge 13 commits intoNixOS:masterfrom
ndl:submit/extra-jax-libs
Closed

Packaged extra JAX libs#152754
ndl wants to merge 13 commits intoNixOS:masterfrom
ndl:submit/extra-jax-libs

Conversation

@ndl
Copy link
Contributor

@ndl ndl commented Dec 30, 2021

Motivation for this change

This PR packages some of the commonly-used JAX frameworks / libraries.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.05 Release Notes (or backporting 21.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@ndl ndl requested review from FRidh and jonringer as code owners December 30, 2021 17:23
@ndl ndl requested a review from samuela December 30, 2021 17:23
@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Dec 30, 2021
@ndl
Copy link
Contributor Author

ndl commented Dec 30, 2021

@samuela I took the liberty of switching dm-tree to source builds in place (the master version now builds with cmake => no need to keep around binary version?) and keeping you in maintainers in the resulting package - I hope that's fine :-)

@ofborg ofborg bot added the 8.has: package (new) This PR adds a new package label Dec 30, 2021
@ofborg ofborg bot requested a review from lebastr December 30, 2021 18:00
@ofborg ofborg bot added 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Dec 30, 2021
@samuela
Copy link
Member

samuela commented Dec 30, 2021

Oh very nice! I myself packaged a few of these on my fork of nixpkgs (https://github.com/samuela/nixpkgs/tree/scratch) but never got around to contributing them upstream.

I think these changes all look good to me broadly speaking. However, my preference is to keep PRs as small as possible. Would you mind sending PRs for each package independently? I know that is a bit more work on your part, but I promise to review and merge them as quickly as possible to keep the process moving along smoothly for you.

@samuela I took the liberty of switching dm-tree to source builds in place (the master version now builds with cmake => no need to keep around binary version?) and keeping you in maintainers in the resulting package - I hope that's fine :-)

Yes, I think that's great!

@ndl
Copy link
Contributor Author

ndl commented Dec 30, 2021

However, my preference is to keep PRs as small as possible. Would you mind sending PRs for each package independently?

Sure, can do that - but note that some packages are inter-dependent (e.g. elegy -> dm-haiku -> jmp or flax -> optax -> chex -> dm-tree), which means somewhat complicated submission process.

What do you think about grouping the dependent packages into PRs? That would mean the following PRs list:

  1. objax
  2. dm-tree + chex + optax + flax
  3. jmp + dm-haiku
  4. deepdish + treeo + treex + fix to tensorboardx + elegy

There's still a dependency from (4) to (2) + (3) but at least that's one dependency and the rest can proceed independently.

Of course, if you still prefer individual packages - we can do that as well, it will just take longer because we need to do many of them sequentially then.

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for large package additions like this, sometimes it's easier if you chunk up the individual packges into smaller PRs. Then we can merge them as they become ready.

Comment on lines +28 to +47
-set(ABSEIL_VER 20210324.2)
-include(ExternalProject)
-ExternalProject_Add(abseil-cpp
- GIT_REPOSITORY https://github.com/abseil/abseil-cpp.git
- GIT_TAG ${ABSEIL_VER}
- PREFIX ${CMAKE_SOURCE_DIR}/abseil-cpp
- CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${CMAKE_SOURCE_DIR}/abseil-cpp
- -DCMAKE_CXX_STANDARD=${CMAKE_CXX_STANDARD}
- -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
- -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}
- -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
- -DCMAKE_POSITION_INDEPENDENT_CODE=${CMAKE_POSITION_INDEPENDENT_CODE}
- -DLIBRARY_OUTPUT_PATH=${CMAKE_SOURCE_DIR}/abseil-cpp/lib
-)
-ExternalProject_Get_Property(abseil-cpp install_dir)
-set(abseil_install_dir ${install_dir})
-include_directories (${abseil_install_dir}/include)
-
+find_package(absl REQUIRED)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't have to be a blocker for this PR, but it might be nice to upstream the package with a find_package ... fallback to external type of workflow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, will see if I can upstream it separately, will keep the patch meanwhile.

src = fetchFromGitHub {
owner = "cgarciae";
repo = "${pname}";
rev = "${version}";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
rev = "${version}";
rev = version;

@samuela
Copy link
Member

samuela commented Dec 30, 2021

Of course, if you still prefer individual packages - we can do that as well, it will just take longer because we need to do many of them sequentially then.

I'd still prefer individual packages, but on these sorts of questions I defer to maintainers that have more experience in nixpkgs than I, eg @jonringer. I know it's annoying in the short term but it really helps to keep the git blame clean and I promise to give you a fast turnaround time on any issues I'm tagged in!

@jonringer
Copy link
Contributor

I'd still prefer individual packages, but on these sorts of questions I defer to maintainers that have more experience in nixpkgs than I, eg @jonringer. I know it's annoying in the short term but it really helps to keep the git blame clean and I promise to give you a fast turnaround time on any issues I'm tagged in!

I've also seen large PRs have many small nitpicks which block the PR. It can be very draining as well to have "0 progress" over a long period

@ndl
Copy link
Contributor Author

ndl commented Dec 31, 2021

Per discussion above, closing in favor of individual PRs. I've migrated all the changes to the relevant commits, so I believe no review comments were lost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants