go2nix: 20160307 -> 0.1.0

[kamilchm]

go2nix compatible with #16017

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • OS X
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

---

[rushmorem]

./result has no bin directory. Seems to be the case now for all go programs I have tested.

[kamilchm]

I use nix-build -A go2nix.bin but you're right it should produce all outputs by default https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/go-modules/generic/default.nix#L198
I'll try to find out how to fix it.
@zimbatm do you have any suggestions?

[kamilchm]

@vcunat if you look at https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/go-modules/generic/default.nix do you see anything that can breaks default output for Go builds?

[zimbatm]

I don't have much to add except that it's going to be annoying for users if they expect to install by the attribute path.

[kamilchm]

I looked back how goPackages work before the rework and for me it looks like there were only the bin attribute that was used as top-level package.
I changed go2nix to target go2nix.bin.
If we agree that this is right I need to change it for other packages from #16017 (comment).
This should keep all packages extracted from goPackages backward compatible but I think we should to fix buildGoPackage to produce all outputs by default.

[rushmorem]

Yes, previously only the bin attribute was included as the top level package because packages that depended on a particular go package could still access the package via the goPackages namespace.

Not being able to install by attribute path will indeed annoy users but I don't think including all outputs in the same derivation is the way to go. I think the current design is technically more superior. @kamilchm I think your work in #16017 was an awesome start. Now I think we just need to bring back the goPackages namespace. However, this time go-packages.nix will have the following expression in it:

go2nix = callPackage ../development/tools/go2nix { };

and all-packages.nix will then have:

go2nix = goPackages.bin;

I think this will give us the following benefits:-

• Better architecture overall. If a package only depends on a particular go library, why bring that library's binaries along as well. Likewise, if we only need the binary, why pull in the source code too
• Namespaces are back, yay! Putting go packages into their own namespace makes it easier to name packages as they won't compete with other packages for the global namespace (ignoring the binaries for now)
• Brings back backwards compatibility for the most part
• Makes it easier to automate management of go packages in future
• This is similar to how most language communities on manage their packages in nix
• Takes us one step closer to matching Haskell's ecosystem
• Makes it easier to tell which go packages are already in nix

[kamilchm]

I'll try to do it tommorow.

[rushmorem]

Awesome, thanks!

[zimbatm]

In what cases do you find yourself re-using the "out" of a binary package ? I would prefer if the goPackage had a flag to say if the package is meant for consumption by other go packages or just as a binary to be installed by the user.

IMO the <lang>Package is a bad form. Not all projects are clearly defined, some use multiple languages. If there are conflicts in package names then we have to resolve them on the top-level anyways. The goPackages was useful when we wanted to share dependencies (explicitly) between executable but we found that to be impractical (because of combinatoric complexity). Haskell is a bit different because they has pre-defined sets of dependencies that work together (and even them are moving away from hosting multiple of those sets).

[kamilchm]

Another though I have is to create 2 separate forms:

1. buildGoApp/buildGoPackage/buildGoProgram/..... that will be used to build apps exposed in all-packages and those will expose only binaries
2. buildGoEnv (or sth) than we can use for Go development enviroments used as nix-shell where we can specify goPackages/goDeps to be available in GOPATH or can be used as part of buildGoApp

What do you think about moving this discussion to the mailing list?

[rushmorem]

@zimbatm the difference between binary packages and libraries is not so clear cut. Some binaries are also libraries. Take vault for example. Its main package is in github.com/hashicorp/vault but it also has a few packages under that which can be easily reused by other binaries and libraries alike.

I totally understand your point about projects that use multiple languages. This is why I'm a big fan of #16017. I agree that stuffing full derivations of all the packages of a particular language in one file is bad and a maintenance nightmare. However, IMHO having a go-packages.nix that only has a bunch of callPackages (like we already do in all-packages.nix anyway) is not so bad.

Projects that are not clearly defined do not have to go under one language namespace. If they have different libraries for different languages then the respective languages can also create packages under their own namespaces that point to the relevant path of that package's source tree. The binary of the package itself can then directly go in all-packages.nix.

I wasn't suggesting that we include multiple derivations of the same package like Haskell does eg lts-5_11, lts-5_12, lts-5_13 etc but that we callPackage from that file like they do. I brought Haskell up because to me it seems like they have the most respected ecosystem on nix as far as language support goes (if posts in the mailing list are anything to go by).

[rushmorem]

@kamilchm moving this discussion to the mailing list sounds good. In the meantime if it's not too much trouble would you care to document how the current go packaging now works so as not to throw people off. I would also like to merge this so we can start using the new go2nix.

[kamilchm]

Yep, I'm taking on docs.
Is change in https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/go.xml will be enough?

[zimbatm]

Alright. In any case I don't want to be holding this PR with my ranting :) Avoiding the combinatoric issues we where having before is the real win for me.

[rushmorem]

@zimbatm I totally agree :)

[rushmorem]

@kamilchm I suppose so, but I'm not entirely sure.

[vcunat]

vcunat if you look at https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/go-modules/generic/default.nix do you see anything that can breaks default output for Go builds?

Well, there is "out" and "bin" in that definition. IIRC nix-env will install "bin" by default. nix-build is a little annoying in this, as it will only create result -> $out symlink normally; I personally often use nix-build -A foo.all which creates symlinks for all outputs, e.g. adds result-bin in this case...

[kamilchm]

back to:

./result has no bin directory
annoying for users

I think we should better describe who are the users.

1. If users are those who use nix-build they are doing package development more than using it. It this case they can nix-build -A foo.all or nix-build -A foo.bin as @vcunat said. It's more about default result from nix-build and it's not an issue for buildGoPackage only.
2. If users are those who use nix-env -i foo we don't need to target foo.bin in all-packages because bin results will be available in nix-profile even now.
3. If users are Go developers that wants to use nix as their vendoring tool IMO we need to write it as separate user story that will include usage of go2nix + nix-shell that I haven't touched yet.

So the steps I can do next:

• document nix-build behaviour in buildGoPackage: updated docs for goPackages after rework #16129 for p. 1
• point all-packages to full buildGoPackage and remove .bin for p. 2
• work on full Go + Nix developer workflow for p. 3

How about that?

CC: @cstrahan