-
-
Notifications
You must be signed in to change notification settings - Fork 15.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Julia build system #208379
Julia build system #208379
Conversation
- Add 'withPackages' and 'juliaPackages' passthru attributes to 'julia_16-bin', 'julia_18-bin', 'julia-bin' and 'julia-stable-bin'. - The 'withPackages' function creates an environment for the Julia compiler with additional packages. - 'build-julia-package.nix' is used to compile Julia packages and artifacts. julia build system: fix typos
e56c973
to
fb5c4b5
Compare
fb5c4b5
to
0f6a870
Compare
…ges and julia2nix
This is super awesome. Is there a way to go from a If there is an accepted way of building this up from a |
If you pass one or more package names to |
I've added an option to |
If there is a |
- Make lib, packages, and help functions from build-julia-package.nix available to Julia packages to facilitate overwriting the buildPhase and add non Julia dependencies. - Remove temporary and log files after build to avoid collisions
Thank you for tackling this! I think it looks great so far, apart from julia.withPackages having no version. I think it would be good to passthrough the version of the initial julia. |
Additionally it seems like Conda does not build, I think this needs a more elaborate workaround, as simply making a python with necessary packages available still leads to failiures
where I already tryed adding
just for clarity: I install SymPy and IJulia not Conda directly. |
Regarding if [[ -f ./deps/build.jl ]]; then
pushd deps
julia -- build.jl
popd
fi the const ROOTENV = "/nix/store/vb5avfbixcljv6422qjyg1m9567jp8c0-julia-bin-1.8.3-Conda-1.7.0/share/julia/conda/3"
const MINICONDA_VERSION = "3"
const USE_MINIFORGE = true
const CONDA_EXE = "/nix/store/vb5avfbixcljv6422qjyg1m9567jp8c0-julia-bin-1.8.3-Conda-1.7.0/share/julia/conda/3/bin/conda" and the build succeeds. You can change {
pname = "Conda";
version = "1.7.0";
src = fetchurl {
url = "https://pkg.julialang.org/package/8f4d0f93-b110-5947-807f-2305c1781a2d/6e47d11ea2776bc5627421d59cdcc1296c058071";
name = "julia-bin-1.8.3-Conda-1.7.0.tar.gz";
sha256 = "1e3ed90c6d374877db3245282ce1e25087e1dd496605801b5ae6483bac0ce528";
};
requiredJuliaPackages = [ JSON VersionParsing ];
preBuild = "export CONDA_JL_HOME=/path/to/conda/env";
} Note that you'll also have to add a miniconda package otherwise it will download a version of it at the first use and error out because that conda will not find I'll push the change later on. |
It turns out that some packages like, e.g.,
I'm leaning for 2. as it seems the most flexible. |
The build part sounds great! |
The honest answer is: I don't know and have no clue how to use it. I tried to build |
…NIX_JULIA_PRECMD The NIX_JULIA_PRECMD environment variable precedes the julia compiler command and allows, for example, to make a frame buffer available to packages that need it at build time by setting said variable to "xvfb-run" (and add xvfb-run to the buildInputs).
ce041f9
to
8854488
Compare
{
description = "try out ImageView using julia2nix";
inputs.flake-utils.url = "github:numtide/flake-utils";
inputs.nixpkgs.url = "github:fedeinthemix/nixpkgs/wip-julia-build-system";
outputs = { self, nixpkgs, flake-utils }:
flake-utils.lib.eachDefaultSystem (system: let
pkgs = nixpkgs.legacyPackages.${system};
fixpkgs = pkg:
if pkgs.lib.strings.hasInfix "Gtk" pkg.pname || pkg.pname == "ImageView" then pkg // { preBuild = ''export NIX_JULIA_PRECMD=${pkgs.xvfb_run}/bin/xvfb-run''; }
else pkg;
in {
devShell = pkgs.mkShell {
nativeBuildInputs = [ ((pkgs.julia-bin.withPackages
(ps: [ ])
(ps: map fixpkgs (import ./env.nix {juliaPkgs = ps; inherit (pkgs) fetchurl; }))
).overrideAttrs (o: {
nativeBuildInputs = o.nativeBuildInputs ++ [ pkgs.wrapGAppsHook ];
dontWrapGApps = true;
postBuild = with pkgs.lib.strings; ''
${removeSuffix "\n" o.postBuild} \
"''${gappsWrapperArgs[@]}"
'';
})) ];
};
});
} I think it might be the missing wrapper with this I can open an image and then zoom, click and move in the frame (all in gnome wayland). For this it might be worth it to have an argument that toggles whether the gappsWrapperArgs should be added to the julia wrapper. (Maybe a package could say it needs gtk and thereby enable the wrapper args) Additionally, do you think it would be possible to implement pre-compilation without using Pkgs, maybe by using each pkg once in the build script? Also, I think it would be nice if julia2nix had a toggle to make the file an importable function, right now I use
|
let nixPackages = (nix juliaPackages); | ||
upstreamPackages = (packagesFromUpstream upstreamPkgsList); | ||
|
||
packagesFromUpstream = ps: builtins.map (p: upstreamPkgs."${p.pname}") ps; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason to handle this using two arguments, instead of checking whether the package is an element of the definitions in nix and if not call juliaPackages.juliaPackagesFromList or export juliaPackages.juliaPackagesFromList and make it more similar to python where it is possible to call python3.withPackages (p: [ (python3Packages.buildPythonPackage) ]
such that the function call would look more like
julia.withPackages (p: [ p.Plots ] ++ (juliaPackages.juliaPackagesFromList (import ./packagelist.nix ...)))
.
But someone more experienced with nix package standards might be better to judge this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to use a single argument first and hit an infinite recursion problem. The packages refer to each other before they are defined. The trick to make it work in a lazy language is to build a fix point, a widely exploited trick in nix. However, if you try to process them, then the evaluation doesn't stop. My solution was to separate the lists so as not having to use them before they get defined. It may be possible to use a single argument, but at the moment I don't know how to do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have an idea either, maybe someone more experienced like @SuperSandro2000 or @bennofs could give an opinion.
I tried your flake and, interestingly, get the exact same error as before. I then updated my system (22.11 release) and I don't get a seg. fault anymore, but when I move the pointer on the picture, it still changes aspect ratio and get other nasty artifacts. This may be related to Gnome Wayland compositor or some other component not playing well with the resolution of my monitor (3440x1440). I recently hit other bugs related to this. Could you try it on your system without the gappsWrapperArgs? If it works it means that it's not necessary.
All packages are compiled. If you look into
I thought about this as well, but for the moment I'll leave on the wish list :-) |
Indeed for me it even works without the wrapper and I don't have any artifacts using gnome wayland (but a resolution of 3840x2160). Looking forward to having this merged! |
It turns out that the segmentation fault was caused by |
f8ad7f8
to
c47be78
Compare
@benneti I've added an option to |
I think I just stumbled over a fundamental problem with the dual approach, i.e. dependency resolution can break, my particular problem right now is that the Plots in this branch depends on Latexify 0.15.17 while SymPy from my personal env pulls in Latexify 0.15.18 which leads to a collision. |
Right now I've used the Package directories approach to code loading for simplicity and to mimic the approach used by the libraries coming with the compiler. To avoid collisions we may have to switch to the "Project-environments" one. I've never used it in a stacked way, but from the documentation it should work. |
In that case this might be the better choice as (at least) two independently generated environments need to be combined. |
By the way, the new arguments are and work great! Thanks for that :) |
I've changed my mind and think it's actually challenging to solve this. Here some thoughts. Top level dependencies are in Julia environments options in nixpkgs:
For the moment I propose to stick with version 1, the current situation. |
Question:is |
I'm not sure what you mean by lock file in this context. The |
@benneti I believe I've found a solution to name clashes using approach 2. (stacked package directories). The package collection in nixpkgs and the upstream one now live in different directories. This way we don't get name collisions between the two. Name clashes in each of the two collections can be avoided by using Now, if you just import a package directly (as previously suggested), Julia will pick the first one found. If both collections have a version of it, the upstream one will be used (without version checks). The trick to pick the right version is to use It would be helpful if you could try if it solves your problem. Note that I had to update the nixpkgs collection, and you'll have to do the same for the upstream ones because I had to add an extra piece of information to the definitions. To get an old version with |
We can simply make Julia packages coming with binary blogs work by using I'm closing this PR and I'll open another one based on this approach and a single argument |
That does not align well with our standards. Programs should be compiled from source if possible inside the sandbox. |
Description of changes
This PR adds a build system for Julia packages and artifacts loosely based on the
octave
one. The main entry point is thewithPackages
function added as apassthru
attribute to Julia compilers. An example usage isThe first argument includes packages included in
nixpkgs
while the second permits to include upstream packages. A suitable definition for the latter is obtained withjulia2nix
.License
I've not found a simple automatic way to determine the license of packages. For this reason the packages included in this PR in the file pkgs/development/julia-modules/default.nix are meant for test purposes. These are all the dependencies for the package
Plots
(counting artifacts separately, almost 200!). This is a notable package currently not working innixpkgs
and patched by the proposed build system. All entries were imported withjulia2nix
.There are some discussions about licenses here
Testing
When you test it, please make sure not to have any of the tested packages in your home depot. One way to avoid problems is to point the first entry in the environment variable
DEPOT_PATH
to an empty temporary directors.Things done
sandbox = true
set innix.conf
? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)nixos/doc/manual/md-to-db.sh
to update generated release notes@ninjin @7c6f434c @NickCao