Build arbitrary Julia package environments in Nixpkgs

[thomasjm]

Description of changes

tl;dr: this PR offers a new approach to build arbitrary Julia environments in Nixpkgs, in the same style as python.withPackages. You can try it like this (using whatever packages you like):

gh pr checkout 225513 # Check out this branch
nix run --impure --expr 'with import ./. {}; julia.withPackages ["Plots" "JSON3"]'

How does it work?

The steps are documented in default.nix. To summarize:

1. Establish a pinned version of the Julia General registry. The version I've made is here and it's been specially preprocessed to add Nix sha256 hashes for every package version; more on this later.
2. Take our list of desired packages and invoke Julia's low-level package resolution function to get a full package closure, with UUIDs and version numbers for every desired package.
3. Generate a .nix file containing fetchgit calls for all the desired packages, leveraging sha256 hashes from the special registry.
4. Import the result of step 3 (IFD)
5. Construct a minimal Julia registry using the results of step 4. This minimal registry contains only the packages we need, and the repo fields have been replaced with on-disk paths in the Nix store.
6. Next, do a similar song and dance for Julia binary artifacts. Scan over all the downloaded packages and extract the artifacts they require, generating another .nix file.
7. Import the result of step 6 and use it to build an artifacts Overrides.toml pointing all the artifacts to local Nix store paths (IFD)
8. Assemble all of the above to create a Julia project folder and a Julia depot, then do a makeWrapper call on Julia to set everything up. You can optionally pass a boolean precompile to control whether everything gets precompiled.
9. Now you have a Nix-packaged Julia environment!

The only downside here is that this process involves IFD in step 4 and step 7. I'm hoping the benefits will outweigh the drawbacks here and the community will be willing to land this. In practical terms, the IFD means that Nix isn't able to process the package downloads and artifacts downloads in parallel, which means it takes a bit longer to run them sequentially. And possibly this isn't testable on Hydra. But still, Julia packaging has been a challenge for years and this gets it done!

If the IFD is a no-go, we could move the pieces around to make this a 2-step process where you first generate a set of .nix files, and then build them yourself, similar to julia2nix. IMO the ergonomics of that are worse though.

A couple other notes:

• I put the full list of Julia package names in package-names.nix. The only purpose of this is to expose them in an attrset, so that it's easy to explore the available packages in a Nix repl. I figure it's not too big of a file, but we could remove it.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[farnoy]

👍 Seems to work, with a non-critical error as it tries to access REPL history. This is x86_64-linux:

$ nix run --impure --expr 'with import (fetchTarball "https://github.com/codedownio/nixpkgs/archive/julia-modules.tar.gz") {}; juliaWithPackages ["Plots" "JSON3"]'
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |

ERROR: SystemError: opening file "/nix/store/8nfb9v9xyay4419b22yb20mizpk4nja2-julia-depot/depot/logs/repl_history.jl": Read-only file system
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base ./error.jl:176
# cut
[ Info: Disabling history file for this session
julia> using Plots

julia>

[thomasjm]

Right, I added a note at the top about giving Julia a writable depot to make that error go away.

[benneti]

I think as of now this approach does not work with PyCall (or PythonCall packages), trying to build for example

`juliaWithPackages [ "SymPy" ]` fails because it cannot install 
┌ Info: Using the Python distribution in the Conda package by default.
└ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
[ Info: Downloading miniconda installer ...
ERROR: LoadError: Could not resolve host: github.com while requesting https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_6>
Stacktrace: ...

for the previous PRs this could be fixed by adding a preBuild like
export PYTHONHOME="${julia-python}" (where julia-python is a python with all necessary packages)
but I don't think this works in a simple way here, because we cannot override the depot (which is where we would need the preBuild environment setup.

[thomasjm]

@benneti it should work now in the latest branch! I had forgotten to pass Python to Julia in the earlier parts of the process. I also added a mapping where we can keep common Julia packages and the Python packages they depend on, so SymPy will work out of the box.

In general, you can pass a custom Python with extra packages installed via override:

nix run --impure --expr 'with import ./. {}; ((juliaWithPackages.override { python3 = python3.withPackages (ps: [ps.foo]); }) ["Foo"])'

[benneti]

Great! This fixes SymPy, there is however another whole class that needs to be treated (and I think actually two cases therein) namely PythonCall packages (like PythonPlot) without any PyCall and with PyCall in the same depot.
In the second caseI use

    export JULIA_CONDAPKG_OFFLINE=yes
    export JULIA_CONDAPKG_BACKEND="System"
    export JULIA_CONDAPKG_EXE=${micromamba}/bin/micromamba
    export PYTHONHOME="${julia-python}" # necessary to find modules
    export JULIA_PYTHONCALL_EXE="@PyCall"

I think not all are strictly necessary, but the last one ist the most important one and should point to python.interpreter if there are no PyCall packages in the depot.

[thomasjm]

Okay, I think I've got it @benneti. I added a map of "package implications," so we can say that PythonCall being present requires us to also install PyCall. This plus some new environment variables makes PythonPlot work.

FWIW, there appear to be about 20 other packages in the General registry which use CondaPkg.jl. I think the principled way to deal with them would be to scan through them looking for CondaPkg.toml files, at the same time we do the artifact scan. But we could also just manually add things to extra-python-packages.nix.

[benneti]

Nice, thanks! The automatic scan would be amazing, but maybe also a bit overkill as a solution.

Last question, is there a reason you don't set-default JULIA_DEPOT_PATH to $HOME/.julia (mind though that I am not sure whether it is possible to use makeWrapper because we need $HOME in " not '?

To make it slightly more clear, why not add this line as the first line of the wrapper:
export JULIA_DEPOT_PATH=${JULIA_DEPOT_PATH-"$HOME/.julia"}

[shivaraj-bh]

How much of an effort will it be to support darwin? I was thinking of using julia-bin package from Nixpkgs instead of julia which provides darwin support as well. On the first try I faced some issue with multiprocessing python library used in extract_artifacts.py, debugging that.

Do you see any problem in using julia-bin instead of julia?

[benneti]

Also I just tried a julia "instantiate" a julia package and realized, that the julia intern package manager is completely broken in juliaWithPkgs even when setting JULIA_DEPOT_PATH, is there a way to get around this?

[thomasjm]

Last question, is there a reason you don't set-default JULIA_DEPOT_PATH to $HOME/.julia

I think that's a good idea! Let me try doing that (with an overridable option to disable it).

Do you see any problem in using julia-bin instead of julia?

Not off the top of my head @sbh69840. It would be great if you could figure out what's happening with multiprocessing on macOS.

the julia internal package manager is completely broken in juliaWithPkgs even when setting JULIA_DEPOT_PATH

Hmm, that's sort of the goal of juliaWithPackages, no? To make a static package environment. FWIW I just tried adding a new package to an environment and it worked fine for me, as long as I first activated a local project first. Otherwise it will try to write to the Project.toml in the Nix store.

[benneti]

Sorry indeed installing works, but precompilation and using fails for me.
I tried with a system julia without 'ExcelFiles' which I installed in a local project.
Precompilation fails and using it leads to another error ERROR: LOAD_PATH entries cannot contain ':'.
I'll try to debug some more tomorrow if you have no immediate idea what the problem might be.

EDIT: Are you sure having JULIA_PROJECT as a ":" separated list is supported if not it might be better to have set-default instead of suffix.

[thomasjm]

Are you sure having JULIA_PROJECT as a ":" separated list is supported

You're right, I just changed it to set-default

precompilation and using fails for me

I had to add a mapping for the Python package xlrd to get ExcelFiles to work via juliaWithPackages.

Trying to add it to a local project seems to work for me, other than the missing Python package. To be clear I'm doing this:

cd $(mktemp -d)
nix run --impure --expr 'with import <nixpkgs> {}; ((juliaWithPackages) ["SymPy"])'
julia> ]
(project) pkg> activate .
(tmp.SwF22zOWgb) pkg> add ExcelFiles
...
(tmp.SwF22zOWgb) pkg> <backspace>
julia> import Pkg
julia> Pkg.precompile()
julia> using ExcelFiles
[ Info: Precompiling ExcelFiles [89b67f3b-d1aa-5f6f-9ca4-282e8d98620d]
ERROR: LoadError: InitError: PyError (PyImport_ImportModule

The Python package xlrd could not be imported by pyimport.
...

It seems like it would work if you injected a Python with xlrd.

[benneti]

Yeah with set-default everything seems to work as expected on the julia side! But I think overriding python3 with a python that has packages installed is ignored, due to recalling withPackages, it would be great if it was possible to add python packages manually.

[thomasjm]

due to recalling withPackages

Yeah it's weird that multiple .withPackages calls don't stack? It seems you can't use an overridden python3.withPackages if you're already passing a Julia package like SymPy that does it.

However, it does work to set PYTHONPATH before launching Julia.

[thomasjm]

Hmm, I think the new default $HOME/.julia depot path causes a problem for precompilation. Now Julia seems to be precompile again when you do using on a package that was already precompiled in the Nix store depot.

[benneti]

is the precompilation happening always? For me it seems to work (at least sometimes) as long as I am not in a Julia project.

[shivaraj-bh]

Not off the top of my head @sbh69840. It would be great if you could figure out what's happening with multiprocessing on macOS.

I will have a look at this, if this works we can support Darwin as well.

[thomasjm]

is the precompilation happening always?

I'm having trouble pinning down exactly what causes it to happen. Julia's always been a little twitchy about what causes additional precompiles. It does seem related to when you've activated your own project, so I'm not going to worry about it.

[shivaraj-bh]

Regarding this: #20649 (comment)
Building the package RDatasets works but as soon as I try to import it, it fails because it is trying to open a read-only file from julia-depot/depot/logs/scratch_usage.toml. Is there any easy way to solve this? Or will this require a change in upstream itself?

[thomasjm]

@sbh69840 I can't reproduce that problem. I think it will work as long as you have a writable depot in JULIA_DEPOT_PATH. The latest code adds ~/.julia to this path if it's empty. Are you using the latest code? (You can also do export JULIA_DEPOT_PATH=$(mktemp -d))

[benneti]

I just realized that while now building the packages works, the artifacts in the local depot are not patched such that a lot of packages don't work.
Do you think it would be possible to have a small flake.nix in a project that reads in the Manifest and Project.toml files of this project?

[thomasjm]

I don't follow, what do you mean "not patched"? Can you provide an example?

[benneti]

for example if I have a typical julia project with its own "Manifest.toml" and "Project.toml" which might contain a certain version of Plots.jl which will pull in the GR manifest.
If I now activate this project it will download the GR artifact but it will not be patched, therefore plotting (using GR) will not work.

This is why the other PR introduced the FHS environemnt, here it would probably make more sense to have a way to make the project into a flake project and have the flake read the Manifest and Project.toml and build the dependencies in the nix store.

[thomasjm]

Hmm, I still don't totally see what you mean by "patched" -- do you mean fixing up dynamic library paths? A precise repro would be helpful.

I do think that activating your own project is borderline out-of-scope for juliaWithPackages. The idea of building a Julia environment with Nix is to provide all the dependencies with Nix up front, so downloading extra artifacts from the Julia package manager is done at your peril. But maybe I'm misunderstanding what you're trying to do.

[benneti]

Nevermind, it is actually trivial to wrap juliaWithPackages in buildFHSUserEnv such that project based package management actually works. It might not be the most nix way, but is simple enough for easy compatibility if necessary.

What I initially meant was what you say (I think), but gluing them more together by allowing something like juliaWithProject that takes in a Manifest.toml and Project.toml and generates the appropriate nix expression.

[benneti]

Sorry for spamming this thread, but there is (hopefully one last change I would suggest), taking a look at https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH the second expression of the default value does not make sense the versioned environment should probably point to the nix store, therefore I would suggest to do
--set-default JULIA_LOAD_PATH '@:${projectAndDepot}/project/Project.toml:@v#.#:@stdlib'
hopefully this will also result in reusing the precompiled packages form the store in projects that depend on the same version.
The order apparently plays a role (i.e. putting the projectAndDepot in the end did not work as expected in my tests).

[SomeoneSerge]

we're not actually exposing any Julia environment derivations at the top level,
callCabal2Nix

👍🏻

I don't think I understand this one. artifactsNix already generates a Nix file which you could open in your editor if you want.

...a .nix file in the /nix/store, if I'm on the same page.

What I mean is that the python script contains some pieces of Nix code and the only way to run a syntax check on is to actually build a sample package. But you could (unless I'm not seeing something) replace those chunks of Nix with a single json.dump (which we trust to produce a valid JSON), and then pass the json contents to some pkgs/development/julia-modules/generic.nix. This generic.nix could then be reviewed (linted, autoformatted) independently, without any builds

[thomasjm]

But you could (unless I'm not seeing something) replace those chunks of Nix with a single json.dump

FWIW what you describe sort of already exists for the Julia sources; see the closureYaml attr. In principle you could take what happens in sources_nix.py to produce julia-sources.nix and do all that processing in Nix--but I wouldn't particularly want to attempt it.

The artifacts side of things is even more complex. I think the refactor you describe could be done, but most of the meat would have to stay in Python (for example, using subprocess to invoke Julia subprocesses). I think you'd end up with an extra step and not much extra clarity to show for it, because generic.nix would be fairly short.

[GTrunSec]

• Be able to construct a Julia environment from a Project.toml/Manifest.toml, rather than a flat list of packages.
• Be able to supply artifact overrides as part of the Nix expression.
• TempestSDR: this package has an issue with OpenGL display on NixOS. We have a workaround earlier in this thread, but it would be good to get it working without fuss.
• CUDA: there is probably a similar issue to the OpenGL one above when it comes to leveraging hardware-specific CUDA libraries.

These motivations should be on the next PR.

At this point, the way I'm feeling about it is that the PR is good and useful for many use-cases, and it would be good to merge it now.

The remaining feature requests and issues are completely valid, but solving them all here could hold up this PR for an indeterminate amount of time. Instead I'd love to tackle them all as extensions and follow-up issues. For example, the artifact overrides issue deserves a design discussion of its own. On a personal note, it would be reassuring to see that all this work isn't going to be for nothing :)

Agreed, I have been using this PR for some time now, and it is time to merge it into nixpkgs. Although there may be some other tasks pending, this PR is already sufficiently refined.

[GTrunSec]

@SuperSandro2000 @NickCao
This PR has been ignored for a long time, and I would appreciate it if a maintainer with merge permissions could take a look and give it some attention. Thank you, @thomasjm, for supporting and contributing to the Julia2nix ecosystem.

[NickCao]

@SuperSandro2000 @NickCao This PR has been ignored for a long time, and I would appreciate it if a maintainer with merge permissions could take a look and give it some attention. Thank you, @thomasjm, for supporting and contributing to the Julia2nix ecosystem.

As previously said, I have little knowledge of the julia ecosystem so might not be able to give constructive suggestions, but I trust your expertise as the maintainer of Julia2Nix.jl. The overall implementation does looks a bit convoluted but do lay a solid foundation for future improvements. So, I do intend to merge this, but after the branch of 23.11, giving it more time to mature before getting into a stable release. And a bit bike shedding: I see that julia is wrapped twice, first as juliaWrapped, then into julia-env, is this a hard requirement of the implementation?

[GTrunSec]

And a bit bike shedding: I see that julia is wrapped twice, first as juliaWrapped, then into julia-env, is this a hard requirement of the implementation?

The first wrapper just merges withPackages attribute into each of julia-*bin/julia (same as passthruFun in python), the juliaWrapEnv is utilized(wrapping the julia)only when you invoke julia_xxx-bin.withPackages ["xx"]. The ambiguity lies in the different implications of mkJuliaEnv and WrapJulia.

[NickCao]

May you add release note entries and some documentation? Then this would be good to merge.

[thomasjm]

Okay @NickCao, I've added some documentation!

Also added a release note but I'm not sure if it's good/in the right place, so please take a look.

[NickCao]

For the document part, cc @NixOS/documentation-team

[fricklerhandwerk]

Docs are very well-written, but please reformat to one sentence per line. There is a style guide on nix.dev which the docs team applies to all documentation we review. 👍

Side note: I'm not a fan of withPackages [ "strings" ], but I have no say here and likely this is bikeshedding, but there is at least the consideration of consistency with Python that you should probably check with the Nixpkgs architecture team (@infinisil since there is still now way to ping the team on GitHub).

No approval here because what matters is the code. Looks like a great contribution overall, thanks a lot!

[thomasjm]

Thanks @fricklerhandwerk!

Side note: I'm not a fan of withPackages [ "strings" ]

FWIW, a previous version of this PR had a full list of package names checked in, to make it possible to present an attrset of packages like Python does. It was decided that it wasn't worth it because it was large (>100KB IIRC) and would require extra maintainance work to keep updated.

[NickCao]

Implementation is complicated, but we can improve that later. The document is easy to follow, played with Plots and it works. Regarding the API design, since we are using IFD and it's expensive to construct the attrset beforehand, I'm happy with the current approach. This is a final call for comments and I will merge in 3 days if no objections.

And once again thank @thomasjm and everyone in this thread for your efforts.