Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Julia build system #208379

Closed
wants to merge 18 commits into from
Closed

Conversation

fedeinthemix
Copy link
Contributor

@fedeinthemix fedeinthemix commented Dec 30, 2022

Description of changes

This PR adds a build system for Julia packages and artifacts loosely based on the octave one. The main entry point is the withPackages function added as a passthru attribute to Julia compilers. An example usage is

pkgs.julia-bin.withPackages
  (ps: with ps; [ Plots ])
  (ps: with ps; [
    { pname = "Tokenize";
      version = "0.5.25";
      src = fetchurl {
        url = "https://pkg.julialang.org/package/0796e94c-ce3b-5d07-9a54-7f471281c624/90538bf898832b6ebd900fa40f223e695970e3a5";
        name = "julia-bin-1.8.3-Tokenize-0.5.25.tar.gz";
        sha256 = "00437718f09d81958e86c2131f3f8b63e0975494e505f6c7b62f872a5a102f51";
      };
    }

    { pname = "CSTParser";
      version = "3.3.6";
      src = fetchurl {
        url = "https://pkg.julialang.org/package/00ebfdb7-1f24-5e51-bd34-a7502290713f/3ddd48d200eb8ddf9cb3e0189fc059fd49b97c1f";
        name = "julia-bin-1.8.3-CSTParser-3.3.6.tar.gz";
        sha256 = "647fc5588cb87362d216401a0a4124d41b80a85618a94098a737ad38ae5786c4";
      };
      requiredJuliaPackages = [ Tokenize ];
    }
  ])

The first argument includes packages included in nixpkgs while the second permits to include upstream packages. A suitable definition for the latter is obtained with julia2nix.

License

I've not found a simple automatic way to determine the license of packages. For this reason the packages included in this PR in the file pkgs/development/julia-modules/default.nix are meant for test purposes. These are all the dependencies for the package Plots (counting artifacts separately, almost 200!). This is a notable package currently not working in nixpkgs and patched by the proposed build system. All entries were imported with julia2nix.

There are some discussions about licenses here

Testing

When you test it, please make sure not to have any of the tested packages in your home depot. One way to avoid problems is to point the first entry in the environment variable DEPOT_PATH to an empty temporary directors.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@ninjin @7c6f434c @NickCao

@ofborg ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Dec 30, 2022
- Add 'withPackages' and 'juliaPackages' passthru attributes to
  'julia_16-bin', 'julia_18-bin', 'julia-bin' and 'julia-stable-bin'.
- The 'withPackages' function creates an environment for the Julia
  compiler with additional packages.
- 'build-julia-package.nix' is used to compile Julia packages and
  artifacts.

julia build system: fix typos
@fedeinthemix fedeinthemix force-pushed the wip-julia-build-system branch from fb5c4b5 to 0f6a870 Compare December 30, 2022 17:33
@github-actions github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog 8.has: documentation This PR adds or changes documentation labels Dec 31, 2022
@fedeinthemix fedeinthemix changed the title Wip julia build system Julia build system Dec 31, 2022
@mayl
Copy link
Contributor

mayl commented Dec 31, 2022

This is super awesome. Is there a way to go from a manifest.tomlto the packages list without resolving and locking all the dependencies manually? Is builtins.fromToml enough?

If there is an accepted way of building this up from a manifest.toml that would be a good addition to the docs.

@fedeinthemix
Copy link
Contributor Author

If you pass one or more package names to julia2nix, it looks up and prints out the definitions for all required dependencies (recursively). No need for manual lookup. In addition behind the scenes it uses Pkg to makes sure that all the listed package versions are compatible with each other.

@fedeinthemix
Copy link
Contributor Author

I've added an option to julia2nix to extract the desired list of packages from a Project.toml file.

@fedeinthemix
Copy link
Contributor Author

If there is a Manifest.toml file in the same directory as the specified Project.toml file, then the first is used. Otherwise a manifest is generated from the latter. The list of packages is then produced from the manifest.

- Make lib, packages, and help functions from build-julia-package.nix
  available to Julia packages to facilitate overwriting the
  buildPhase and add non Julia dependencies.

- Remove temporary and log files after build to avoid collisions
@benneti
Copy link
Contributor

benneti commented Jan 3, 2023

Thank you for tackling this! I think it looks great so far, apart from julia.withPackages having no version. I think it would be good to passthrough the version of the initial julia.

@benneti
Copy link
Contributor

benneti commented Jan 3, 2023

Additionally it seems like Conda does not build, I think this needs a more elaborate workaround, as simply making a python with necessary packages available still leads to failiures


@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/klmhiq8kb67p6h4ndyq2c9kqb2pwqgy7-julia-bin-1.8.3-Conda-1.7.0.tar.gz
source root is /build/Conda
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "buildPhase" }
building
ERROR: LoadError: Conda is not properly configured.  Run Pkg.build("Conda") before importing the Conda module.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] top-level scope
   @ /nix/store/j5693zd4r7gpdyw2kh9vb8n00xyz3blc-julia-bin-1.8.3-Conda-1.7.0/share/julia/packages/Conda/src/Conda.jl:26
 [3] include
   @ ./Base.jl:419 [inlined]
 [4] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_>
   @ Base ./loading.jl:1554
 [5] top-level scope
   @ stdin:1
in expression starting at /nix/store/j5693zd4r7gpdyw2kh9vb8n00xyz3blc-julia-bin-1.8.3-Conda-1.7.0/share/julia/packages/Conda/src/Conda.jl:1
in expression starting at stdin:1
ERROR: Failed to precompile Conda [8f4d0f93-b110-5947-807f-2305c1781a2d] to /nix/store/j5693zd4r7gpdyw2kh9vb8n00xyz3blc-julia-bin-1.8.3-Conda-1.7.0/share/juli>
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:1707
 [3] compilecache
   @ ./loading.jl:1651 [inlined]
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1337
 [5] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [6] macro expansion
   @ ./loading.jl:1180 [inlined]
 [7] macro expansion
   @ ./lock.jl:223 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144

where I already tryed adding

let  julia-python = python3.withPackages (ps:
    with ps; [
      sympy jupyter
    ]);
in ...
        pname = "Conda";
        ...
        buildInputs = [ julia-python ];
        preBuild = ''
        export PYTHONPATH="${julia-python}/${julia-python.sitePackages}"
        export PYTHON="${julia-python.interpreter}"
        '';
...

just for clarity: I install SymPy and IJulia not Conda directly.

@fedeinthemix
Copy link
Contributor Author

Regarding deps.jl. If I include the following snippet in the buildPhase

    if [[ -f ./deps/build.jl ]]; then
      pushd deps
      julia -- build.jl
      popd
    fi

the deps.jl file is generated. with the following content

const ROOTENV = "/nix/store/vb5avfbixcljv6422qjyg1m9567jp8c0-julia-bin-1.8.3-Conda-1.7.0/share/julia/conda/3"
const MINICONDA_VERSION = "3"
const USE_MINIFORGE = true
const CONDA_EXE = "/nix/store/vb5avfbixcljv6422qjyg1m9567jp8c0-julia-bin-1.8.3-Conda-1.7.0/share/julia/conda/3/bin/conda"

and the build succeeds. You can change ROOTENV by defining CONDA_JL_HOME as in

    {
        pname = "Conda";
        version = "1.7.0";
        src = fetchurl {
            url = "https://pkg.julialang.org/package/8f4d0f93-b110-5947-807f-2305c1781a2d/6e47d11ea2776bc5627421d59cdcc1296c058071";
            name = "julia-bin-1.8.3-Conda-1.7.0.tar.gz";
            sha256 = "1e3ed90c6d374877db3245282ce1e25087e1dd496605801b5ae6483bac0ce528";
        };
        requiredJuliaPackages = [ JSON VersionParsing ];
        preBuild = "export CONDA_JL_HOME=/path/to/conda/env";
    }

Note that you'll also have to add a miniconda package otherwise it will download a version of it at the first use and error out because that conda will not find /lib64/ld-linux-x86-64.so.2.

I'll push the change later on.

@fedeinthemix
Copy link
Contributor Author

It turns out that some packages like, e.g., Gtk expect to be able to connect to a frame buffer at build time. The suggested solution for those packages seems to be to use xvfb-run, see JuliaGraphics/Gtk.jl#346. For that we need to change the compile line in the buildPhase from
julia -e '...' to xvfb-run julia -e '...'. To do that we have 3 options

  1. Leave it to the package author to redefine the buildPhase
  2. Define a switch or an environment variable like NIX_JULIA_PRECMD to easily add it.
  3. Leave xvfb-run in for all packages. It looks like it does no harm when not needed.

I'm leaning for 2. as it seems the most flexible.

@benneti
Copy link
Contributor

benneti commented Jan 3, 2023

The build part sounds great!
For the second part, do Gtk.jl apps work then without the wrapper args (I had a wrapped julia using wrapGAppsHook)?

@fedeinthemix
Copy link
Contributor Author

The build part sounds great! For the second part, do Gtk.jl apps work then without the wrapper args (I had a wrapped julia using wrapGAppsHook)?

The honest answer is: I don't know and have no clue how to use it. I tried to build ImageView and tried to run the example on the home page https://github.com/JuliaImages/ImageView.jl.
I can display an image, but then as soon as I move the pointer on the image it seg. fault :-( I have no clue if this is related to the fact that I'm running on Wayland or not. The error is the same as discussed here swaywm/wlroots#2032.

…NIX_JULIA_PRECMD

The NIX_JULIA_PRECMD environment variable precedes the julia compiler
command and allows, for example, to make a frame buffer available to
packages that need it at build time by setting said variable to
"xvfb-run" (and add xvfb-run to the buildInputs).
@fedeinthemix fedeinthemix force-pushed the wip-julia-build-system branch from ce041f9 to 8854488 Compare January 3, 2023 19:45
@benneti
Copy link
Contributor

benneti commented Jan 4, 2023

{
  description = "try out ImageView using julia2nix";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:fedeinthemix/nixpkgs/wip-julia-build-system";
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = nixpkgs.legacyPackages.${system};
      fixpkgs = pkg:
        if pkgs.lib.strings.hasInfix "Gtk" pkg.pname || pkg.pname == "ImageView" then pkg // { preBuild = ''export NIX_JULIA_PRECMD=${pkgs.xvfb_run}/bin/xvfb-run''; }
        else pkg;
    in {
      devShell = pkgs.mkShell {
        nativeBuildInputs = [ ((pkgs.julia-bin.withPackages
          (ps: [ ])
          (ps: map fixpkgs (import ./env.nix {juliaPkgs = ps; inherit (pkgs) fetchurl; }))
          ).overrideAttrs (o: {
            nativeBuildInputs = o.nativeBuildInputs ++ [ pkgs.wrapGAppsHook ];
            dontWrapGApps = true;
            postBuild = with pkgs.lib.strings; ''
              ${removeSuffix "\n" o.postBuild} \
              "''${gappsWrapperArgs[@]}"
            '';
          })) ];
      };
    });
}

I think it might be the missing wrapper with this I can open an image and then zoom, click and move in the frame (all in gnome wayland). For this it might be worth it to have an argument that toggles whether the gappsWrapperArgs should be added to the julia wrapper. (Maybe a package could say it needs gtk and thereby enable the wrapper args)

Additionally, do you think it would be possible to implement pre-compilation without using Pkgs, maybe by using each pkg once in the build script?

Also, I think it would be nice if julia2nix had a toggle to make the file an importable function, right now I use

      echo '{ juliaPkgs, fetchurl }:' > ./env.nix
      echo 'with juliaPkgs;' >> ./env.nix
      julia2nix ImageView >> ./env.nix

let nixPackages = (nix juliaPackages);
upstreamPackages = (packagesFromUpstream upstreamPkgsList);

packagesFromUpstream = ps: builtins.map (p: upstreamPkgs."${p.pname}") ps;
Copy link
Contributor

@benneti benneti Jan 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to handle this using two arguments, instead of checking whether the package is an element of the definitions in nix and if not call juliaPackages.juliaPackagesFromList or export juliaPackages.juliaPackagesFromList and make it more similar to python where it is possible to call python3.withPackages (p: [ (python3Packages.buildPythonPackage) ]
such that the function call would look more like
julia.withPackages (p: [ p.Plots ] ++ (juliaPackages.juliaPackagesFromList (import ./packagelist.nix ...))).
But someone more experienced with nix package standards might be better to judge this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to use a single argument first and hit an infinite recursion problem. The packages refer to each other before they are defined. The trick to make it work in a lazy language is to build a fix point, a widely exploited trick in nix. However, if you try to process them, then the evaluation doesn't stop. My solution was to separate the lists so as not having to use them before they get defined. It may be possible to use a single argument, but at the moment I don't know how to do it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have an idea either, maybe someone more experienced like @SuperSandro2000 or @bennofs could give an opinion.

@fedeinthemix
Copy link
Contributor Author

I think it might be the missing wrapper with this I can open an image and then zoom, click and move in the frame (all in gnome wayland). For this it might be worth it to have an argument that toggles whether the gappsWrapperArgs should be added to the julia wrapper. (Maybe a package could say it needs gtk and thereby enable the wrapper args)

I tried your flake and, interestingly, get the exact same error as before. I then updated my system (22.11 release) and I don't get a seg. fault anymore, but when I move the pointer on the picture, it still changes aspect ratio and get other nasty artifacts. This may be related to Gnome Wayland compositor or some other component not playing well with the resolution of my monitor (3440x1440). I recently hit other bugs related to this.

Could you try it on your system without the gappsWrapperArgs? If it works it means that it's not necessary.

Additionally, do you think it would be possible to implement pre-compilation without using Pkgs, maybe by using each pkg once in the build script?

All packages are compiled. If you look into .../share/julia/compiled you'll see files for all of them. Sometimes I also see further compilation when I import a package. This may be due to it introducing a function specialization for a function defined in another package.

Also, I think it would be nice if julia2nix had a toggle to make the file an importable function, right now I use

I thought about this as well, but for the moment I'll leave on the wish list :-)

@benneti
Copy link
Contributor

benneti commented Jan 5, 2023

Indeed for me it even works without the wrapper and I don't have any artifacts using gnome wayland (but a resolution of 3840x2160).

Looking forward to having this merged!

@fedeinthemix
Copy link
Contributor Author

fedeinthemix commented Jan 7, 2023

It turns out that the segmentation fault was caused by ImageView not finding the system fonts. See JuliaImages/ImageView.jl#278

@fedeinthemix fedeinthemix force-pushed the wip-julia-build-system branch from f8ad7f8 to c47be78 Compare January 7, 2023 14:41
@fedeinthemix
Copy link
Contributor Author

@benneti I've added an option to julia2nix to generate a function. I've also added options to generate a flake.nix or shell.nix and make it easy to spawn a Julia with packages. Before merging this PR, the nixpkgs reference in those files will have to be modified to point to this branch.

@benneti
Copy link
Contributor

benneti commented Jan 10, 2023

I think I just stumbled over a fundamental problem with the dual approach, i.e. dependency resolution can break, my particular problem right now is that the Plots in this branch depends on Latexify 0.15.17 while SymPy from my personal env pulls in Latexify 0.15.18 which leads to a collision.
Maybe the better solution for now would be to only support the generated env to ensure a coherent environment, the downside would be that we do not have the binary cache :(.

@fedeinthemix
Copy link
Contributor Author

I think I just stumbled over a fundamental problem with the dual approach, i.e. dependency resolution can break, my particular problem right now is that the Plots in this branch depends on Latexify 0.15.17 while SymPy from my personal env pulls in Latexify 0.15.18 which leads to a collision. Maybe the better solution for now would be to only support the generated env to ensure a coherent environment, the downside would be that we do not have the binary cache :(.

Right now I've used the Package directories approach to code loading for simplicity and to mimic the approach used by the libraries coming with the compiler. To avoid collisions we may have to switch to the "Project-environments" one. I've never used it in a stacked way, but from the documentation it should work.

@benneti
Copy link
Contributor

benneti commented Jan 10, 2023

In that case this might be the better choice as (at least) two independently generated environments need to be combined.

@benneti
Copy link
Contributor

benneti commented Jan 10, 2023

By the way, the new arguments are and work great! Thanks for that :)

@fedeinthemix
Copy link
Contributor Author

I've changed my mind and think it's actually challenging to solve this. Here some thoughts.

Top level dependencies are in Project.toml. The full dependency graph (dependencies of dependencies) are in Manifest.toml. These files are maintained by the package manager 'Pkg' not by the Julia compiler itself. The latter is "only" a consumer of those files. For 'Pkg' to work, it needs access to package registries, the General one
available at pkg.julialang.org. Generation of Project.toml is straightforward, generation of Manifest.toml is much less so.

Julia environments options in nixpkgs:

  1. Single "Package Directory" (current approach)

    • Cons:
      • May get package collisions between upstream and nixpkgs provided packages.
    • Pros:
      • Simple
      • Collisions are explicit. Can easily resolve them by dropping some nixpkgs provided packages.
  2. Use stacked "Package Directories"

    • Cons:
      • A package may (possibly silently) load a dependency differing (version) from what it expects. (versions are only considered by 'Pkg', not Julia itself.)
    • Pros:
      • No directory name collisions.
  3. Maintain a copy of the General registry in nixpkgs and use 'Pkg' to generate 'Project.toml' and 'Manifest.toml'.

    • Cons:
      • have to regularly update it
    • Pros:
      • Can use "Project Environments" and use 'Pkg' at environment generation.
  4. Use julia2nix to create Project.toml and Manifest.toml on top
    of the list of packages.

    • Cons:
      • Much less user-friendly: need to generate and pass to nixpkgs 3 files instead of 1.
      • How to handle sub-sets of nixpkgs provided packages?
    • Pros:
      • No General registry in nixpkgs.
      • No collisions
  5. Patch 'Pkg' and only rely on it.

    • Cons:
      • Invasive
      • Need to maintain patch
    • Pros:
      • Standard Julia development flow.

For the moment I propose to stick with version 1, the current situation.

@pasqui23
Copy link
Contributor

Question:is manifest.toml a lockfile?

@fedeinthemix
Copy link
Contributor Author

fedeinthemix commented Jan 11, 2023

Question:is manifest.toml a lockfile?

I'm not sure what you mean by lock file in this context. The Manifest.toml file holds the entire dependency graph of a Julia project. It is generated and maintained by the Julia package manager Pkg and fixes the versions of all packages in a project.

@fedeinthemix
Copy link
Contributor Author

@benneti I believe I've found a solution to name clashes using approach 2. (stacked package directories). The package collection in nixpkgs and the upstream one now live in different directories. This way we don't get name collisions between the two. Name clashes in each of the two collections can be avoided by using julia2nix.

Now, if you just import a package directly (as previously suggested), Julia will pick the first one found. If both collections have a version of it, the upstream one will be used (without version checks). The trick to pick the right version is to use Pkg and create (or activate) an environment for the local project and add the packages in it. This way Pkg does check the version and will instruct Julia to load the appropriate one. I've instructed Pkg to
work "offline" so as to prefer local packages. If necessary, it will still download the General registry. If you already have a copy of the registry, and you get an error about a missing package version, you may have to update the registry manually (registry update). (One can still disable the offline mode with import Pkg; Pkg.offline(false), but you may get non-working packages directly from pkg.julialang.org).

It would be helpful if you could try if it solves your problem. Note that I had to update the nixpkgs collection, and you'll have to do the same for the upstream ones because I had to add an extra piece of information to the definitions. To get an old version with julia2nix you can use the Pkg syntax, e.g., julia2nix [email protected].

@fedeinthemix
Copy link
Contributor Author

We can simply make Julia packages coming with binary blogs work by using buildFHSUserEnv. With it the second argument of withPackages looses its raison d'être and the workflow is much streamlined.

I'm closing this PR and I'll open another one based on this approach and a single argument withPackages. Thanks to everyone who helped out testing and point out problems.

@SuperSandro2000
Copy link
Member

That does not align well with our standards. Programs should be compiled from source if possible inside the sandbox.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog 8.has: documentation This PR adds or changes documentation 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants