WIP: splitting packages into multiple outputs

[vcunat]

(descriptions will be updated in-place)

Current status

• Most packages build fine: KDE4, some GNOME3 packages, Xfce. Hydra jobsets: http://hydra.nixos.org/jobset/nixos/closure-size http://hydra.nixos.org/jobset/nixpkgs/closure-size
• I'm using NixOS off this (almost) without any problems since October, and there seem to be a few other users.
• Preliminary introduction: http://lipa.ms.mff.cuni.cz/~cunav5am/nix/closure-size-notes.pdf (slightly outdated ATM).

Current merge blocker:

• See WIP: splitting packages into multiple outputs #7701 (comment).

Minor stuff (not blocking merge to master)

• linking problems in rare cases stdenv: principial linking problems when putting static and dynamic libraries into different directories #12085
• a few icons stopped appearing.
• Darwin seems untested, and many other platforms might have problems.
• Handling of outputs installed by default. Merging PRs: nixos systemPackages: rework default outputs #12653, nix-env: respect meta.outputsToInstall nix#815, and perhaps further improvements.

[vcunat]

Opened a PR so people can browse the differences more easily, and we get a place to discuss the topic. Some hoped to have this ready for the upcoming release, but I don't seem to be able to stabilize it enough all by myself.

[vcunat]

I hope the key to making NixOS boot is in understanding these messages:

systemd[722]: Failed at step CHDIR spawning /nix/store/i1k1qs79ib9674mgdh9vw6xjbzxlhyb5-coreutils-8.23/bin/rm: No such file or directory

I don't understand that – the executable does exist on that path, and that path pkgs.coreutils is (still) in the default list of packages to be visible by systemd services. Moreover, only pkgs.coreutils.info output was added in this branch, as the rest are just the executables and locales (and man pages, but we typically want them and they are relatively small).

[copumpkin]

👍 really exciting to see this happening, thanks!

[copumpkin]

Is there a single VM test or simple command that someone can run to reproduce/iterate on the failures you're encountering?

[vcunat]

I think any attempt to boot NixOS would fail, even in a VM, but I haven't tried. Some of my filesystems won't mount IIRC (it's just plain btrfs).

[copumpkin]

This might be silly, but would it ease the transition if we took the existing mkDerivation and made it so that multiple-output derivations had a "synthetic" (lazily generated, obviously) outPath that acted like a buildEnv of the multiple outputs?

[vcunat]

Not much AFAIK. The point is getting rid of dependencies on the other files. If I knew what particular packages are causing the problems, I could e.g. just make them single-output again, for the time being. I've done a few such (partial) steps back for now, when there were too many references that would need changing and the space benefit was low.

[lucabrunox]

@vcunat perhaps about your mounting problems, you could try not splitting the mount.* related packages, or btrfs util related packages etc..

[copumpkin]

Is there any progress? Is this going to make it into 15.06?

[vcunat]

No, we would need several weeks at least to stabilize that in any case (it's a rather disturbing change), so I postponed it past the release (focusing more on that ATM).

[copumpkin]

Fair enough. Another thing I was wondering is whether there's a more incremental approach to merging this in, where we can slowly make the harmless packages multi-outputs in small bites without trying to do so much at once. In my experience, monolithic changes tend to languish because they're too overwhelming to fix/review/work on.

Basically, is there a sensible way to break this PR into smaller less-risky units that we could start merging in sooner?

[vcunat]

Well, maybe. I might somehow drop lots of the old commits and focus just e.g. on stdenv stuff first, or whatever packages have best savings/risk ratio. There were lots of packages split years ago, but the changes seem untested properly (and/or rotten w.r.t. master changes in the meantime).

[domenkozar]

I've cherry-picked a5fb184 into staging as it makes Python closure for ~100MB lighter

[vcunat]

TODO note:

vcunat: so .gir are only useful during build, right?
Lethalman: vcunat, yes
vcunat: (I wondered about .gir when working on multiple-output splitting)
Lethalman: .gir are needed to create .typelib
Lethalman: and because a .gir include another .gir, you need them in XDG_DATA_DIRS to be found
Lethalman: but at runtime, only the .typelib is needed

[copumpkin]

Perhaps we should cherry-pick more of this into master, updating the PR branch as we go along to show the diff getting smaller? It seems dauntingly large right now, which is a maintenance nightmare given an evolving nixpkgs, and also scares people from contributing. Are the commits mostly self-contained, or do they actually depend on one another? I'd be willing to pick a few low-hanging fruit in the name of helping this along.

[vcunat]

They often aren't self-contained by themselves, and none of the "commit states" in the branch yields a fully functioning system, I believe.

I expect to go back to it after the release and start by porting the shared infrastructure to staging (mostly autotools-focused ATM), which by itself should be relatively easy (for me) and breaking very few packages (with fixes somewhere in the branch already). And perhaps write some description/documentation/tutorial.

After that, anyone will be able to use it to split packages they understand. When I wanted to stabilize this, I was plagued by the scope of breakage in packages that were split and broken by it (e.g. systemd+udev proved to be painful).

[dezgeg]

So I took a look at this last week (branch at https://github.com/dezgeg/nixpkgs/commits/multiple-outputs-v2) and while the merge conflicts were indeed pretty bad, this isn't a totally lost cause (it does boot in Virtualbox for me, and at least closures.{smallContainer,xfce,ec2} can be built). Off the top of my head, these are the breakages I noticed:

• udev can't find it's rules provided by the package (see commit msg of dezgeg@04bd57b for details). This probably explains the problems mounting filesystems.
• some KDE/Qt packages can't locate PCRE
• gcj fails to build due to cycles between outputs
• libsodium doesn't compile on ARM (linking fails due to bad relocation)
• cross compilation is totally broken; the cross GCC fails to build

[vcunat]

@dezgeg: thank you! I'm wading through the branch again, and I do find your commits useful. I'll be integrating them.

You haven't resolved the udev/systemd issues, right? I'm thinking of unsplitting it and perhaps using a separately built libudev in dependencies or something.

[dezgeg]

Nope, I didn't even try fixing udev/systemd.

[vcunat]

So, now I'm running my NixOS on this branch!

• The boot problems were probably fixed by a changed approach to systemd/udev b44d846.
• As noted above, some KDE packages were failing to locate pcre or automoc, but I just added those.
• libsodium had problems even on x86_64-linux, but not anymore c83b591.
• Many other small build problems got fixed.

The switch to gcc-5 (and others) are queued for evaluation on staging now, so I don't expect staging this PR before we stabilize that a bit. Also, I merged latest systemd branch into this one (due to possible interactions with that udev rework).

[vcunat]

I think this branch is stable enough to go to master, so I'll resolve conflicts and merge it to staging now. That's because there's a security mass rebuild in staging that hasn't started rebuilding yet, and I don't want to build both mass rebuilds separately. People wanting to avoid breakages can switch to 16.03...

[deepfire]

: -)

[domenkozar]

I cancelled the closure-size jobset on hydra. Thanks!

On Sun, 10 Apr 2016, 10:42 Kosyrev Serge, notifications@github.com wrote:

: -)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#7701 (comment)

[lucabrunox]

It's happening........... thanks a lot for the hard work.

[abbradar]

I don't want to spam this thread with congratulations but I can't keep myself from doing this ~_^. Thank you very much -- and now we can share the burden of fixing what broke and also enjoy the benefits of this wonderful work!

[jagajaga]

Great job!!

[abbradar]

On Hydra closure-size branch is not yet cancelled (or at least looks so) -- should it be like this?

[vcunat]

The closure-size branch contains the same tree as staging, so it won't really matter. I had cancelled all scheduled builds when I merged (on staging and closure-size).

[jtojnar]

How does libxml2Python work?

{
libxml2Python = pkgs.buildEnv { # slightly hacky
    name = "libxml2+py-${self.libxml2.version}";
    paths = with libxml2; [ dev bin py ];
    inherit (libxml2) passthru;
    # the hook to find catalogs is hidden by buildEnv
    postBuild = ''
      mkdir "$out/nix-support"
      cp '${libxml2.dev}/nix-support/propagated-build-inputs' "$out/nix-support/"
    '';
  };
}

Is not it enough to add libxml2.py to propagatedBuildInputs of python package?

Why does not the postBuild fail? dev output is not listed in propagatedBuildOutputs.

[vcunat]

@jtojnar: I think that buildEnv hack was needed at that time, but I could have been wrong. I was really swamped by all the changes (splitting something like a hundred packages). It will certainly be better if it works without the hack.

dev is the default output for build inputs. And explicitly referenced paths are put into build-time closure anyway...