buildGoModule: Fix overriding with overlay-style stdenv by ShamrockLee · Pull Request #225051 · NixOS/nixpkgs

ShamrockLee · 2023-04-06T21:32:47Z

Description of changes

The Go package builder buildGoModule is refactored to adopt the overlay-style stdenv.mkDerivation (#119942) and to fix overriding. One can now override vendorHash using overrideAttrs. This addresses #86349 (comment)

The goModules and vendorHash attributes are moved out of passthru as they are already exposed as stdenv.mkDerivation attributes. They can still be accessed as pet.vendorHash, but no longer pet.passthru.vendorHash. The builder now only adds go to passthru.

To address the issue of vendorHash = null through overrideAttrs may break the overriding of goModules, this PR also provides an overridable passthru.overrideModAttrs that will be applied to goModules when vendorHash is not null.

Closes #86349

Comparison to similar PRs:

In this PR, the diff of pkgs/build-support/go/module.nix is shorter than #171586.

Things done

zowoq

It may not be a breaking change but I think we're to close to branch off for this so I'm a very strongly against including it in 23.05.

Also, I am planning to make a few changes to buildGoModule after the branch, e.g. drop vendorSha256 and only have vendorHash, it'll be easier and make this PR simpler if that happens first.

cc @NixOS/golang

ShamrockLee · 2023-04-06T23:39:10Z

It may not be a breaking change but I think we're too close to branch off for this so I'm strongly against including it in 23.05.

Agree. Should I drop change to the the release note?

I place them there just because the new release note hasn't come out yet.

ShamrockLee · 2023-04-07T00:09:11Z

Would it be okay to just remove those mod*Phase attributes? We alredy have overrideModAttrs, which itself will also become a syntax sugar after this chage. Exposing those mod*Phase would be confusing.

zowoq · 2023-04-07T00:17:03Z

Should I drop change to the the release note?

I place them there just because the new release note hasn't come out yet.

Fine to leave them there for now, can be moved once the new release note is created after branch off.

Would it be okay to just remove those mod*Phase attributes? We alredy have overrideModAttrs, which itself will also become a syntax sugar after this chage. Exposing those mod*Phase would be confusing.

Maybe, have to give it some thought first.

Also, I'd appreciate it if you would revert all of the unrelated formatting changes, it makes the diff much harder to review.

ShamrockLee · 2023-04-08T20:43:43Z

Also, I'd appreciate it if you would revert all of the unrelated formatting changes, it makes the diff much harder to review.

It turns out that the indentation difference is not just caused by nixpkgs-fmt, but the inherent difference of two structures.

Even then, I managed to make them align by making reasonable adjustment to the original expression (mainly by simplifying the go-modules part before running nixpkgs-fmt). The adjustment (packed inside the first 2 commits) doesn't bring any rebuild. Maybe I could file another PR containing them to make this one easier to review.

The last commit contains the actual change. The relation between the original implementation and the new implementation can be seen from the diff. The expected attributes are organized at the beginning of the expression, and the corresponding comment documentation are prefixed with ATTRIBUTES:.

zowoq · 2023-04-08T23:08:19Z

@ShamrockLee I'm switching this back to draft as it isn't going to be merged until after branch off.

pkgs/build-support/go/module.nix

zowoq · 2023-04-09T04:04:10Z

#225371

Still needs more testing but I've opened a PR to (mostly) get rid of vendorSha256.

ShamrockLee · 2023-04-09T20:20:33Z

Go packages built with this buildGoModule implementation seems to take noticable longer to evaluate. Not sure if it also happens to #171586.

How can we benchmark the evaluation time?

doronbehar · 2024-08-01T14:18:10Z

doc/languages-frameworks/go.section.md

Sorry for causing the confusion.

No worries :).

In my imagination, the first overriding targeting goModules might be buried somewhere inside the codebase, while another user made another overriding and set vendorHash = null.

The purpose of this documentation is to avoid the conflicts between these overridings.

Hmm this is a bit complicated for at this point, so I'm still not sure I understand. Let's ask this: Why not:

Suggested change

let

pet-foo = pkgs.overrideAttrs (

finalAttrs: previousAttrs: {

passthru = previousAttrs.passthru // {

overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs (

finalModAttrs: previousModAttrs: {

# goModules-specific overriding goes here

postBuild = ''

# Here you have access to the `vendor` directory.

substituteInPlace vendor/github.com/example/repo/file.go \

--replace-fail "panic(err)" ""

'';

}

);

};

}

);

pet-vendored = pet-foo.overrideAttrs { vendorHash = null; };

in

pet-vendored

let

pet-foo = pkgs.overrideAttrs (

finalAttrs: previousAttrs: {

# TODO: Explain this?

vendorHash = null;

passthru = previousAttrs.passthru // {

overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs (

finalModAttrs: previousModAttrs: {

# goModules-specific overriding goes here

postBuild = ''

# Here you have access to the `vendor` directory.

substituteInPlace vendor/github.com/example/repo/file.go \

--replace-fail "panic(err)" ""

'';

}

);

};

}

);

in

pet-foo

Let's avoid the confusion by removing vendorHash = null and leaving it as a simple example demonstrating the correct overriding approach.

doronbehar

Excellent work! Thanks for working on this @ShamrockLee . I will give a few days for @zowoq a few more days to respond to this (it seemed they didn't like it initially), and if there will be no reply I'll merge this.

Fix overriding of vendorHash and various attributes via the fixed point attribute support of stdenv.mkDerivation. Pass as derivation attributes goModules, modRoot, vendorHash, deleteVendor, and proxyVendor. Move goModules and vendorHash out of passthru. Co-authored-by: Doron Behar <doron.behar@gmail.com>

ShamrockLee · 2024-08-11T05:39:25Z

The package in the overriding example in go.section.md was still named pet-foo. I renamed it to pet-overridden.

Haven't responded to any of the new changes happened to the PR recently.

nixos/doc/manual/release-notes/rl-2411.section.md

rl-2411.section.md: fix typo from #225051

philiptaron · 2024-08-20T13:47:38Z

The nixpkgs-manual fails to build due to this change:

nixpkgs/doc/languages-frameworks/go.section.md

Line 94 in bbbd473

### Overriding `goModules` (#buildGoModule-goModules-override)

The parentheses ought to be curly braces. I've opened PR #336061 to fix it.

nixos-discourse · 2024-08-21T12:12:30Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/overrideattrs-and-go-query/50919/2

SuperSandro2000 · 2024-09-02T15:54:22Z

One of my overlays broke because of a mistake I did but I think we can avoid that. Fix in #339042

ShamrockLee · 2024-09-14T14:01:46Z

@doronbehar I proposed lib.toExtension in #336414, addressing the concern in #225051 (comment)

nixos-discourse · 2025-02-01T10:04:12Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-it-possible-to-override-cargosha256-in-buildrustpackage/4393/22

ShamrockLee changed the title ~~goModule: Fix overriding with overlay-style stdenv and add missing parentheses~~ goModule: Fix overriding with overlay-style stdenv Apr 6, 2023

ShamrockLee changed the title ~~goModule: Fix overriding with overlay-style stdenv~~ buildGoModule: Fix overriding with overlay-style stdenv Apr 6, 2023

zowoq previously requested changes Apr 6, 2023

View reviewed changes

ShamrockLee force-pushed the go-module-overlay-stdenv branch 2 times, most recently from eeb5844 to 0e87a61 Compare April 8, 2023 20:40

ShamrockLee marked this pull request as ready for review April 8, 2023 20:49

ShamrockLee requested review from Mic92 and kalbasit as code owners April 8, 2023 20:49

ShamrockLee force-pushed the go-module-overlay-stdenv branch from 0e87a61 to bd74d74 Compare April 8, 2023 21:11

ShamrockLee requested a review from zowoq April 8, 2023 21:13

zowoq marked this pull request as draft April 8, 2023 23:08

zowoq reviewed Apr 9, 2023

View reviewed changes

pkgs/build-support/go/module.nix Outdated Show resolved Hide resolved

pkgs/build-support/go/module.nix Outdated Show resolved Hide resolved

zowoq added the 2.status: work-in-progress label Apr 9, 2023

zowoq self-assigned this Apr 9, 2023

ShamrockLee force-pushed the go-module-overlay-stdenv branch from bd74d74 to fee7aae Compare April 10, 2023 19:41

doronbehar reviewed Aug 1, 2024

View reviewed changes

ShamrockLee force-pushed the go-module-overlay-stdenv branch from bea7e1a to 1d618a3 Compare August 10, 2024 13:01

ShamrockLee requested a review from doronbehar August 10, 2024 13:05

doronbehar approved these changes Aug 10, 2024

View reviewed changes

ShamrockLee force-pushed the go-module-overlay-stdenv branch from 1d618a3 to eed069a Compare August 11, 2024 05:38

doronbehar merged commit 0d920a9 into NixOS:staging Aug 16, 2024

jian-lin reviewed Aug 16, 2024

View reviewed changes

nixos/doc/manual/release-notes/rl-2411.section.md Show resolved Hide resolved

ShamrockLee mentioned this pull request Aug 16, 2024

rl-2411.section.md: fix typo from #225051 #335200

Merged

13 tasks

jian-lin added a commit that referenced this pull request Aug 16, 2024

Merge pull request #335200 from ShamrockLee/release-note-typo

04d11e1

rl-2411.section.md: fix typo from #225051

qezz mentioned this pull request Aug 18, 2024

Go replace take 2 qezz/nixpkgs#1

Draft

philiptaron mentioned this pull request Aug 20, 2024

nixpkgs-manual: fix build #336061

Merged

2 tasks

Aleksanaa mentioned this pull request Aug 21, 2024

go-modules: simplify overriding #171586

Closed

13 tasks

ShamrockLee mentioned this pull request Aug 21, 2024

lib.fixedPoints.toExtension: init #336414

Merged

13 tasks

ShamrockLee mentioned this pull request Feb 2, 2025

treewide: use writableTmpDirAsHomeHook #378110

Merged

13 tasks

ShamrockLee mentioned this pull request Feb 18, 2025

buildRustPackage: support fixed-point arguments via lib.extendMkDerivation #382550

Merged

13 tasks

9999years mentioned this pull request Feb 26, 2025

rustPlatform.buildRustPackage: support finalAttrs style #194475

Closed

13 tasks

ShamrockLee mentioned this pull request Mar 19, 2025

netbird: unbreak netbird-ui build #390989

Merged

13 tasks

zowoq removed their assignment Apr 5, 2025

ShamrockLee mentioned this pull request Aug 17, 2025

buildGoModule: warn for passthru.overrideModAttrs lost after overriding #434546

Merged

13 tasks

ShamrockLee mentioned this pull request Sep 10, 2025

buildGoModule: reference allowGoReference from finalAttrs #441836

Merged

13 tasks

ShamrockLee mentioned this pull request Nov 2, 2025

buildGoModule: intersect given and available meta.platforms #457308

Open

13 tasks

-let
-  pet-foo = pkgs.overrideAttrs (
-    finalAttrs: previousAttrs: {
-      passthru = previousAttrs.passthru // {
-        overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs (
-          finalModAttrs: previousModAttrs: {
-            # goModules-specific overriding goes here
-            postBuild = ''
-              # Here you have access to the `vendor` directory.
-              substituteInPlace vendor/github.com/example/repo/file.go \
-                --replace-fail "panic(err)" ""
-            '';
-          }
-        );
-      };
-    }
-  );
-  pet-vendored = pet-foo.overrideAttrs { vendorHash = null; };
-in
-pet-vendored
+let
+  pet-foo = pkgs.overrideAttrs (
+    finalAttrs: previousAttrs: {
+      # TODO: Explain this?
+      vendorHash = null;
+      passthru = previousAttrs.passthru // {
+        overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs (
+          finalModAttrs: previousModAttrs: {
+            # goModules-specific overriding goes here
+            postBuild = ''
+              # Here you have access to the `vendor` directory.
+              substituteInPlace vendor/github.com/example/repo/file.go \
+                --replace-fail "panic(err)" ""
+            '';
+          }
+        );
+      };
+    }
+  );
+in
+pet-foo

Uh oh!

Conversation

ShamrockLee commented Apr 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

zowoq left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShamrockLee commented Apr 6, 2023

Uh oh!

ShamrockLee commented Apr 7, 2023

Uh oh!

zowoq commented Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShamrockLee commented Apr 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zowoq commented Apr 8, 2023

Uh oh!

Uh oh!

Uh oh!

zowoq commented Apr 9, 2023

Uh oh!

ShamrockLee commented Apr 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doronbehar Aug 1, 2024

Choose a reason for hiding this comment

Uh oh!

ShamrockLee Aug 10, 2024

Choose a reason for hiding this comment

Uh oh!

doronbehar left a comment

Choose a reason for hiding this comment

Uh oh!

ShamrockLee commented Aug 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

philiptaron commented Aug 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixos-discourse commented Aug 21, 2024

Uh oh!

SuperSandro2000 commented Sep 2, 2024

Uh oh!

ShamrockLee commented Sep 14, 2024

Uh oh!

nixos-discourse commented Feb 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

ShamrockLee commented Apr 6, 2023 •

edited

Loading

zowoq left a comment •

edited

Loading

zowoq commented Apr 7, 2023 •

edited

Loading

ShamrockLee commented Apr 8, 2023 •

edited

Loading

ShamrockLee commented Apr 9, 2023 •

edited

Loading

ShamrockLee commented Aug 11, 2024 •

edited

Loading

philiptaron commented Aug 20, 2024 •

edited

Loading