Rework goPackages

[kamilchm]

Motivation for this change

This is my try on better goPackages as I mentioned in #13819 (comment)

I started with an assumption that there's no need to have goPackages that aren't used in a compiled program. So I've created new derivations in proper program categories for all all-packages.nix that comes from go-packages.nix using prototype tool go2nix.
All program dependencies are extracted to json file than can include other json dependency files for better package reuse and making overrides possible.
libs.json and deps.json files can be easily updated with tools like proposed in #13819 or generated from go native dependency managers.

There are few more things to do, but I'm looking for feedback to proritize tasks. Let's make go packaging better.

CC: @edolstra @cstrahan @copumpkin @viric @hrdinka @wkennington @zimbatm

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • OS X
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

---

[zimbatm]

+100. Let's merge this soon before it gets stale.

[offlinehacker]

nice :)

[kamilchm]

I'm working on replacing all fetchgit with fetchFromGitHub. I don't know why but fetchgit doesn't work with version tags on travis :/
Give me an hour ;)

[zimbatm]

Should go2nix be patched to automatically generate the right thing ?

[kamilchm]

I left changes in go2nix until we settle on dependencies file format and buildGoPackage.goDeps attribute.

[kamilchm]

Is it ok like this or maybe we want to produce every otutput? I couldn't figure it out and make 2 styles of it to make it consistent later.

[kamilchm]

strange thing happens on travis:

unpacking source archive /nix/store/mjyfcvkr9dw9in717ya5r4va30nyz9h4-consul-v0.6.4-src
source root is consul-v0.6.4-src
patching sources
patching script interpreter paths in ./ui/scripts/dist.sh
./ui/scripts/dist.sh: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/p7cn4rd7rkc4x0g3354aj5ynh90pfcv2-bash-4.3-p42/bin/bash "
configuring
no configure script, doing nothing
building
/run/user/1001/nix-build-consul-ui-0.6.4.drv-0/consul-v0.6.4-src/ui /run/user/1001/nix-build-consul-ui-0.6.4.drv-0/consul-v0.6.4-src
sh: /run/user/1001/nix-build-consul-ui-0.6.4.drv-0/consul-v0.6.4-src/ui/scripts/dist.sh: Permission denied

I can't reproduce it on my machine :/
What could be wrong?

[groxxda]

Most certainly a travis issue you don't have to care about See

[kamilchm]

So I leave travis as is.

[kamilchm]

Trying again after @domenkozar fix

[kamilchm]

Needs update after #16042

[dguibert]

By getting rid of go-packages.nix, rclone has removed (introduced by #15852).

I wanted to use go2nix save to rebuild the derivation. But how can I populate GOPATH with all the dependencies for rclone (on NixOS)?

[kamilchm]

It should be as easy as using go get github.com/ncw/rclone as stated in http://rclone.org/install/, then go to $GOPATH/src/github.com/ncw/rclone, execute go2nix save and you should see 2 files there default.nix and deps.json. If something goes wrong it'll be go2nix bug that you can report there.

[kamilchm]

@edolstra is there a way to use nix parser as a library? I can generate nix instead of json using text templates in Go but I can't automate package reuse in go2nix without parsing nix.
I remember that there was an initiative by @zimbatm to make a subset of nix with only literal values, but it looks like it's ended as JSON anyway https://github.com/ryantrinkle/nixpkgs/blob/be30ba8e0eabbc2483122ddf9d5911484df3a578/pkgs/build-support/fetchgit/nix-prefetch-git#L326 ?

[zimbatm]

@kamilchm I think we should drop the libs.json altogether. Then it's not a problem because you don't have to parse the nix code, you just generate new deps.nix files.

Because of the content-addressable store, if two programs have the same dependency it will already result in the same derivation in the store. Git also has compression in the pack files so it won't make much difference to duplicate the dependencies on disk. And finally most users will just use the binary builds so it won't make any differences to them.

[kamilchm]

You're right. I'll make a proposal PR with deps.nix and leave the package reuse at nix store level.

[copumpkin]

@kamilchm did you lose all the fetchFromGitHub calls in this change? And the buildFromGitHub call was quite nice, since the vast majority of Go packages have a ton of nearly identical boilerplate and fetch from github.

[copumpkin]

@kamilchm @zimbatm I don't think sharing is a storage-level concern for most of us, so the content-addressable deduplication doesn't really address my concern with it. It's a human-level concern for me, where I want to make sure that if I find that I needed to patch some particular Go package for use with Nix (perhaps because it assumed silly things about an FHS filesystem), I don't have to do that every time I use it, but can do that once. Or that if a particular Go package ends up having a security vulnerability, that I don't need to go hunt down a thousand libs.json files to figure out who's vulnerable.

So in some sense I'm kind of disappointed that this is now the approach, and wish I'd paid attention while this was happening.