Coredns add plugins

[KFearsoff]

Description of changes

Solves #146603

Usage:

coredns-fanout = pkgs.coredns.override {
  externalPlugins = ["fanout"];
  vendorSha256 = "<sha>";
};

Practically speaking, it's not possible to know vendorSha256 ahead of time in this case. Just set it to vendorSha256 = "";. The build will fail, but it will tell you the necessary sha256

This is not elegant, but I haven't found a better way to do it. In case you want to try solving this puzzle yourself, here's a rundown of what needs to be done:

1. Plugins should be appended to plugin.cfg in the source files
2. They should be downloaded via go get
3. Sources should be updated with go generate

I'm not sure how to do it better, since point 2 requires us to have internet access, which is only possible if we use fixed-output derivations (which is what buildGoModule does already), but that would mean we'd have to do the weird dance with vendorSha256

The weird dance is needed because of #86349

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[winterqt]

A few things:

1. This should show the use of pinned plugin versions to ensure reproducibility.
2. Additionally, to improve UX, you should introduce FOD hash validation by checking the go.sum files.

All in all, I'm not really sure how I feel about this. I think it could work, but ideally we'd do something like I did in #14671 (comment), but that has its own issues.

wishes for WebAssembly plugins to avoid all of this pain

[KFearsoff]

1. This should show the use of pinned plugin versions to ensure reproducibility.

I don't quite understand. Do you want to see plugins being able to be pinned? If so, then yeah, that would be good. I'll think of ways how to do that.

2. Additionally, to improve UX, you should introduce FOD hash validation by checking the go.sum files.

I'm afraid I'm not confident enough in my Go package management to understand how that would be done. That would be really great to have, though!

All in all, I'm not really sure how I feel about this. I think it could work, but ideally we'd do something like I did in #14671 (comment), but that has its own issues.

I think you solution looks cleaner, yeah. I think implementing it is harder, and once again, I'm not quite confident enough in my Go package management. We'd also have to write update scripts for CoreDNS plugins if we hope to have that usable, which I've never done and I'm not sure how easy it is. With some help I'm thinking we'll manage to implement it nicely.

wishes for WebAssembly plugins to avoid all of this pain

Oh yeah, there is a lot of pain.

[winterqt]

I don't quite understand. Do you want to see plugins being able to be pinned? If so, then yeah, that would be good. I'll think of ways how to do that.

It's all in how you specify the package to go get (so just what you pass to externalPlugins would be affected), see https://go.dev/ref/mod#modules-overview.

I'm afraid I'm not confident enough in my Go package management to understand how that would be done. That would be really great to have, though!

I can work on an example when I get a chance.

I'm going to mark this as a draft since there's still some uncertainty here (if you disagree, let me know).

[de11n]

Thanks for working on this. We are also in need of this feature. However, as written, this design is not reproducible. I think we need to find a solution that at least tries to maintain reproducibility, even if it can't guarantee it. That would probably require pinning each plugin separately and somehow injecting it into the vendored coredns src right before the build.

[bct]

The initial go-modules derivation fails to build for me (in installPhase) with mv: target directory 'vendor': No such file or directory.

It works if I add go mod vendor to the end of modBuildPhase.

[bct]

#256710 rebases this PR and adds the ability to pin a specific version of each plugin.