goPackages: make it easier to automate updates

[cstrahan]

This is a new PR based on the old #13701, which merged in ee3b295 and reverted in 7760d6e (see there for commentary re: process).

To restate what this does:

What this does

This moves revisions and checksums to a separate, machine readable (specifically, JSON) file.

Why?

Presently, we have to go chasing each dependency of each go package in order to update revisions. This requires opening a browser, going to each GitHub repo, looking at the commits and tags, think long and hard about which revision to choose, open a terminal and get the sha256, then plug the revision and sha256 back into the go-packages.nix file. I tried updating the serf package by hand, and that took almost 3 hours.

Ideally, a tool would provide a central way to pull the last couple commits and tags from each GitHub repo, and either completely automatically update the package set (or a given subset, like serf and it's closure), or it would provide a UI for users that presents the revisions from upstream so you can make intelligent decisions about which revision to use (e.g. there are 4 new commits on master, but version 1.0.0 was tagged just a couple days ago... I think I'll go with tag).

However, this commit doesn't place a dependency on such tooling (in fact, none exists as of yet), so we can consider the design later.

Why JSON, and not Nix?

The Node and Haskell package sets are good examples of where we can update the whole package by regenerating the Nix expressions from scratch. That works for those ecosystems, because they have a culture of tagging releases. They also explicitly state their dependencies and the version bounds thereof. So, with minimal tweaks a separate file (e.g. pkgs/development/haskell-modules/configuration-common.nix) can extend the few troublesome packages with patches and such.

However, that doesn't work for Go packages. Go projects don't explicitly list their dependencies and version bounds. Go projects don't tag releases[1]. If we want to have any chance of updating Go packages in an assisted or automatic way, we need to know what revisions we currently package.

What this doesn't do

• This does not place a dependency on a particular auto-packaging tool.
• This does not make it any harder to update packages by hand.
• This does not change the currently packaged versions.
• This does not introduce massive rebuilds.
• This does not change any interfaces in a backwards incompatible way.
• This does not break any existing packages.

---

[1]: If you're unfamiliar with the Go ecosystem, I repeat: they don't tag releases. They don't have any sort of package repository, so they all pull from each other's master and hope everything works out.

[cstrahan]

GitHub won't show anything in the "Files Changed" tab, so if you want to review the changes without pulling this down and running git diff, you can view the diff here:

https://github.com/NixOS/nixpkgs/pull/13819.patch

[vcunat]

Maybe it's better to view the individual files in github's UI, or some unusual tool (perhaps word-diff).

[cstrahan]

Maybe it's better to view the individual files in github's UI, or some unusual tool (perhaps word-diff).

Yeah, you can also look at the individual files and compare with the parent commit. But I figure it's much easier to see what exactly is going on by just looking at the patch. Conceptually, this is a really simple, mechanical change (in fact, the JSON was dumped by a program).

[vcunat]

Oh, I see. I only looked at the beginning of the patch and I thought it looked like first adding the whole of the new file and the removing the whole of the old file. Now I see I was wrong.

[cstrahan]

/cc @edolstra @copumpkin @viric @hrdinka

Feedback in the positive, negative or passive would be greatly appreciated.

[copumpkin]

and @wkennington

[cstrahan]

From @edolstra on 7760d6e:

---

Yeah, sorry for being a bit abrupt about reverting this, but I've learned it's better to revert quickly when dealing with commits that have a potentially high impact. (E.g. last year wkennington introduced some new derivation style that IMHO wasn't a good idea, and when I finally decided to revert that, it took me half a day to untangle the web of commits that depended on those changes. In this case there were already two commits depending on this commit. Since go-packages.nix contains hundreds of packages, )

Note I'm not necessarily opposed to this commit, but it's kind of a "strategic" change that requires some careful thought, for these reasons:

• It introduces the notion of writing packages in JSON format rather than as Nix expressions. I don't know if that's something we want to promote. In any case it's a new derivation style, and those should always be carefully considered since they make it harder for people to understand / modify a package.
• It introduces yet another auto-update mechanism, and we already have a lot of those, most of them only used by the person who introduced them. (E.g. gnupdate, the .upstream files, language-specific mechanisms...) This too raises the burden for someone else to modify a package because they have to figure out how the particular auto-updater for that package works. We should really come up with a universal auto-update tool, or at least a standard for what they should do (e.g. update Nix expressions in place or overwrite an external version file?).
• (Slightly tangential) It further entrenches go-packages.nix, which I already objected to when it was introduced. We don't have a giant c-packages.nix for everything written in C either. Having one big file is suboptimal e.g. due to the risk of conflicts between different PRs, inability to do git log to see the history of an individual package, etc.

The meta-problem is of course that we don't have a good process for dealing with such changes. Hopefully #13602 will help a bit with that.

[cstrahan]

@edolstra

It introduces yet another auto-update mechanism, and we already have a lot of those, most of them only used by the person who introduced them. (E.g. gnupdate, the .upstream files, language-specific mechanisms...) This too raises the burden for someone else to modify a package because they have to figure out how the particular auto-updater for that package works.

With what I propose, we won't lose the ability to continue updating packages as we currently do, that is, manually. When a tool is created, I would definitely want to see it documented, so people could benefit from it. Unlike the Haskell case, however, we could continue to update packages by hand without having them clobbered and rolled back upon running an update tool (which is something I've had happen long ago when I first tried contributing something Haskell related).

It introduces the notion of writing packages in JSON format rather than as Nix expressions.

I could make the format Nix, and just parse (or evaluate) the file in the (as of yet hypothetical) tool I suggest using to update package revisions.

This too raises the burden for someone else to modify a package because they have to figure out how the particular auto-updater for that package works. We should really come up with a universal auto-update tool, or at least a standard for what they should do (e.g. update Nix expressions in place or overwrite an external version file?).

If it's possible to pull off a single, universal updater, then I agree -- that sounds great. However, I think that's a monumental effort, vs creating a few specialized tools, and providing really good documentation on how to use them (which is precisely what I intend to do once I've written such a tool, but that comes next). I would argue that the burden of needing to know how to use a tool for automated or assisted updates far outweighs the present burden of updating Go packages.

It further entrenches go-packages.nix, which I already objected to when it was introduced. We don't have a giant c-packages.nix for everything written in C either.

Speaking of burden, I think there's an important difference to keep in mind between Go and C packages. The release cycle for C libs usually spans something between months and years; the go release cycle is effectively each commit -- if you don't keep up with changes, your stuff is broken. One change in a single library causes a cascade of required version bumps. C libs have versioned releases; Go packages don't.

Trying to keep up with changes by hand is going to become an intractable ordeal as we package more Go packages. I wanted to update serf, consul and vault for a presentation I planned to give at work at the beginning of this week (totally missed that goal), and updating serf alone took over 2 hours of my time. As is evidenced by my previous work updating other C libs and apps, the C ecosystem usually requires just a couple minutes of effort to find the latest version, update the version in the nix expression (which is usually sufficient to update the url), do a quick nix-prefetch-url -A somePackage.src and plug the resulting sha256 back into the expression. In the go world, I would literally draw a pentagram on the ground and shower myself in goat's blood if it would somehow let me update Go packages that quickly.

[cstrahan]

I suppose that, as an alternative to this PR, I could write my own tool that parses the go-packages.nix, computes the new revs and hashes, and then edits the file in place. That would be quite a bit of work, and it would likely be a tool that only I could use (I probably wouldn't open source it, so as not to confuse people), but I suppose it would be the least controversial: whether the packages I update are done by hand or automated, the results would be identical, so as far as anyone is concerned, it would be like I just did the update manually. What I cant do, however, is spend multiple hours each week trying to keep things up to date.

[wkennington]

You could try using my updater. Its not precise at adding new dependencies
or removing old ones, but it will update everything
On Thu, Mar 10, 2016 at 12:01 PM Charles Strahan notifications@github.com
wrote:

@edolstra https://github.com/edolstra

It introduces yet another auto-update mechanism, and we already have a lot
of those, most of them only used by the person who introduced them. (E.g.
gnupdate, the .upstream files, language-specific mechanisms...) This too
raises the burden for someone else to modify a package because they have to
figure out how the particular auto-updater for that package works.

With what I propose, we won't lose the ability to continue updating
packages as we currently do, that is, manually. When a tool is created, I
would definitely want to see it documented, so people could benefit from
it. Unlike the Haskell case, however, we could continue to update
packages by hand without having them clobbered and rolled back upon running
an update tool (which is something I've had happen long ago when I first
tried contributing something Haskell related).

It introduces the notion of writing packages in JSON format rather than as
Nix expressions.

I could make the format Nix, and just parse (or evaluate) the file in the
(as of yet hypothetical) tool I suggest using to update package revisions.

This too raises the burden for someone else to modify a package because
they have to figure out how the particular auto-updater for that package
works. We should really come up with a universal auto-update tool, or at
least a standard for what they should do (e.g. update Nix expressions in
place or overwrite an external version file?).

If it's possible to pull off a single, universal updater, then I agree --
that sounds great. However, I think that's a monumental effort, vs creating
a few specialized tools, and providing really good documentation on how to
use them (which is precisely what I intend to do once I've written such a
tool, but that comes next). I would argue that the burden of needing to
know how to use a tool for automated or assisted updates far outweighs the
present burden of updating Go packages.

It further entrenches go-packages.nix, which I already objected to when it
was introduced. We don't have a giant c-packages.nix for everything written
in C either.

Speaking of burden, I think there's an important difference to keep in
mind between Go and C packages. The release cycle for C libs usually spans
something between months and years; the go release cycle is effectively
each commit -- if you don't keep up with changes, your stuff is broken. One
change in a single library causes a cascade of required version bumps. C
libs have versioned releases; Go packages don't.

Trying to keep up with changes by hand is going to become an intractable
ordeal as we package more Go packages. I wanted to update serf, consul
and vault for a presentation I planned to give at work at the beginning
of this week (totally missed that goal), and updating serf alone took
over 2 hours of my time. As is evidenced by my previous work updating other
C libs and apps, the C ecosystem usually requires just a couple minutes of
effort to find the latest version, update the version in the nix expression
(which is usually sufficient to update the url), do a quick nix-prefetch-url
-A somePackage.src and plug the resulting sha256 back into the
expression. In the go world, I would literally draw a pentagram on the
ground and shower myself in goat's blood if it would somehow let me update
Go packages that quickly.

—
Reply to this email directly or view it on GitHub
#13819 (comment).

[cstrahan]

@wkennington Where is your updater hosted?

[wkennington]

https://github.com/triton/triton/blob/master/pkgs/development/tools/godep/updatePackageAndDeps.sh
On Thu, Mar 10, 2016 at 12:18 PM Charles Strahan notifications@github.com
wrote:

@wkennington https://github.com/wkennington Where is your updater
hosted?

—
Reply to this email directly or view it on GitHub
#13819 (comment).

[cstrahan]

@wkennington Thanks

[cstrahan]

I'm edging towards just writing my own personal tool to update go-packages.nix in place. I'll have to modify hnix to support source locations, and then I'll have to write a pretty printer that preserves unmodified source locations, and calculate the diffs between source locations, and partially evaluate the parsed expressions so I can resolve things like string interpolations when I analyze existing package definitions...

But it would work. And it would get us out of the predicament that we're currently in; if you look at go-packages.nix, it's seen very few updates over the past 6-12 months. For the very few packages that do push git tags for releases, we're behind by many major revisions (consul, serf, vault, etc), and the other packages are 6-12 months behind master.

Either way, I need to get unblocked. I need to use a bunch of Hashicorp's stuff for some upcoming projects, and I don't want to put in 10 hours updating it all if I'm just going to put in another 10 hours every two to four weeks. And, ideally, everyone gets to benefit from whatever I do. Nothing is more crushing than knowing someone else has put in the hours to do something after you've gone through the exact same steps, and you realize that -- had they contributed their work -- you could have spent an hour or two with friends, instead of doing boring, uninspiring, monotonous grunt work in front of a computer screen.

[kamilchm]

Is this will only work for github repos? What we should do with projects hosted on bitbucket and others then?
How can we handle projects that specifies revisions of its dependecies with tools like godep or glide (eg. vault)?

[cstrahan]

@kamilchm

Hey, thanks for taking a look at this :).

Is this will only work for github repos? What we should do with projects hosted on bitbucket and others then?

As this stands, this just attempts to factor out the GitHub repos, as they represent the overwhelming majority of Go packages (there are only 4 non-GitHub packages, while there are 344 GitHub packages). I would also be fine with extending this to support other hosts, and I'm open to suggestions for how to do so (i.e. what should go-packages.json look like?).

How can we handle projects that specifies revisions of its dependecies with tools like godep or glide (eg. vault)?

That's actually something that I think needs to be discussed. For big projects that use godep to vendor their packages (and there will likely be more vendored projects when 1.6 lands in the next, due to the experimental vendoring system being turned on by default).

Personally, within the context of Go packaging, I feel a _lot_ more comfortable using the vendored packages, as they have been vetted by the authors. I greatly appreciate your work on go2nix for this reason. However, despite my personal feelings on the matter, this PR attempts to address the problem of managing a single set of non-vendored packages - the reason being that I think (or at least thought) that getting something like this accepted would be more likely than getting a whole bunch of vendored packages into Nixpkgs (I figured at some point that would raise some red flags, and then we'd be stuck waiting to hear about how to package really important things like hologram, vault, packer, etc...).

_So some questions remain:_

• What are the guidelines for packaging Go stuff?
• If upstream vendors their dependencies, do we provide a package separate from e.g. go-packages.nix?
• Do we not use vendored dependencies?

Personally, this makes sense to me:

• If an app provides vendored dependencies, use go2nix to generate a package containing the entire closure, and expose that in all-packages.nix.
• For all go projects provide a unified set of packages that build - and hopefully work - together.

Whatever we do, I think we must keep in mind that the perfect is the enemy of the good. The Go community have yet to settle on any meaningful packaging/versioning story, and given the built-in support for vendoring coming up in Go 1.6, I doubt that's going to change any time soon.

[copumpkin]

On vendoring, I think it's undesirable in the context of Nix, as much as it's appealing from the "developer chose the versions" angle. That's probably a whole separate can of worms to discuss in another thread, as it's broader than just Go, but it also applies heavily to Go as you say.

[cstrahan]

@copumpkin I completely agree about it being undesirable. I'm torn, really.

Regardless, I want to reiterate that this PR is focused on how to improve the current state of affairs, where we try to provide single, curated set of Go packages.

Our Go stuff is woefully outdated, and I want an effective way to keep stuff up to date. I will soon be recording a professional-grade, free instructional course on Nix/NixOS/NixOps, and I need this stuff updated before I can finish the course outline and start recording.

[zimbatm]

Why can't we just generate nix expressions from Godeps.json files for each project ?

Maintaining one big set of working packages is not practical for any language except Haskell because they already do all the work. It's an exponential problem that's made worse by the fact that each commit is a potential candidate in go (but it's also unpractical for all the other languages IMO).

Especially for go it doesn't make much difference on the closure size since all themost dependencies are statically linked anyways.

At least by using the project's Godeps.json we will be using the exact same set of dependencies that they vetted works with the project. It means that if we have any issue to report to upstream we will also be much more credible. I think that's quite important if we want to be perceived as a reliable platform.

[kamilchm]

I have one big thing (that may fail) in mind and I hope that I find some time to challenge it next week. I want to start from the roots and extract all goPackages from all-packages.nix, then automatically create separate derivations for all of them with some format of declaring its dependencies (json,nix,..). I hope that I could merge the same dependecies from different top level projects to make final go-packages.{nix|json} then.

But I also agree with @cstrahan that the perfect is the enemy of the good. So if this PR could help us to update current packages quickly, I would merge it. As @cstrahan said it won't make it harder to update packages by hand but gives us a possibility to automate updates.

[kamilchm]

I tried to generate complete separate derivations for few big packages with go2nix. I encountered one error with Consul that needs one more change in buildGoPackage #14493. I can try to extract common dependencies for all generated packages with go2nix then.

[kamilchm]

I took me a while, but I have basic PoC of go packages rework that I'm thinking about.
It's related to https://github.com/kamilchm/go2nix that can be used for package updates then.
This is example of moving 2 apps exposed in all-packages from go-packages to separate top level derivations with reuse of Go dependencies defined in one json file than can be updated automatically kamilchm#2

Few thing you can see there:

1. Deis and Gawp derivations was generated by go2nix save, each having its all dependencies in separate deps.json files and using it instead of extraSrcs.
2. go2nix merge was used go2nix merge deps.json development/go-modules/libs.json to move dependencies to common libs.json and include it in deps files from step one.
3. Each derivation deps file include dependencies from common libs.json, but it can also define its own or override any dependency from libs.json.

Many things needs too be done, like using fetchFromGitHub and go2nix update for updates of dependencies without touching nix derivations or tests and code refactoring. But I want to know if I'm going in the right direction?

[kamilchm]

@cstrahan what do you think about my proposal?

[Mic92]

this pull request has been superseded.