bazel_8: init at 8.1.1 by boltzmannrain · Pull Request #362414 · NixOS/nixpkgs

boltzmannrain · 2024-12-06T13:42:28Z

Started with copying bazel_7 and adapted it, notable changes:

bazelDeps now builds on Darwin in sandbox=true which was previously failing on power management tweaks
using lndir -silent for less build logs
had to disable module_deps feature to avoid getting .. does not depend on a module exporting .. errors on clang/macos importing intrinsics
on macos adding -fno-aligned-allocation on sdk <10.13 to avoid build error saying it isn't supported on <10.13
placing under pkgs/by-name to pass CI, but still calling from pkgs/top-level/all-packages.nix as not sure how to pass darwin sdk and other settings in by-name way

Things done

Add a 👍 reaction to pull requests you find important.

infinisil · 2025-01-20T18:41:06Z

ci/OWNERS

Let's not burden people with more pings unless they confirm being okay with it. @Profpatsch do you even still want to help maintain Bazel? If not, you should probably be removed from here.

And independent of whether @Profpatsch stays listed here, I'm suggesting you add yourself @boltzmannrain, since you've done so much great Bazel maintenance recently :D

timothyklim · 2025-03-05T05:55:15Z

Since PR created, bazel updated to https://github.com/bazelbuild/bazel/releases/tag/8.1.1

seh · 2025-03-05T15:29:20Z

The description includes the following item:

had to disable module_deps feature to avoid getting "... does not depend on a module exporting ..." errors on clang/macos importing intrinsics

Where does that occur within the current patch?

seh · 2025-03-05T15:32:22Z

Where does that occur within the current patch?

Oh, perhaps you meant module_maps, which occurs within the pkgs/by-name/ba/bazel_8/package.nix file .

seh · 2025-03-05T15:48:20Z

pkgs/by-name/ba/bazel_8/package.nix

I found that for what's been ailing me trying to build the Abseil library within Bazel on macOS, just including --host_features=-module_maps in my .bazelrc finally got it all to work.

Thank you so much, Dmitry!

boltzmannrain · 2025-03-06T08:16:40Z

Updated to 8.1.1, removed ci/OWNERS change, applied couple more build workarounds that showed up. ~~Darwin builds have a new unaddressed -faligned-allocation (opposite flag is already added, but maybe not in all contexts)~~ (fixed)

timothyklim · 2025-03-19T11:34:59Z

Can it be merged before 8.2.0?

philiptaron · 2025-03-19T13:07:19Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 362414

`x86_64-darwin`

✅ 1 package built:

bazel_8

philiptaron

I read through this PR as a committer who

a. doesn't use Bazel
b. doesn't know Java that well
c. is just looking to unblock what looks like a stuck PR

So please bear that in mind.

I have a couple big pieces of feedback that would have made it much easier to say "merge immediately".

Splitting the mega-derivation into chunks, each of which was callPackaged from the main one. That makes it possible for me to say "yeah, that checks out" in chunks, rather than trying to keep the whole 600+ Nix derivation in my head as I'm reading it.
Use of patches throughout, so that all of the sed calls were reviewable through seeing them in context. Yes, this will be bigger in terms of bytes. Trying to figure out what a multi-line sed call is going to do is just super hard.
Use of the new Darwin SDK and interaction with recent developments there. More broadly, a lot of this derivation is dedicated to changing Bazel behavior on Darwin, and I found myself wondering whether it was overfitting on a stdenv that no longer is recommended.

There are lots of smaller things I commented on as a read through this PR, but those were the big ones.

I don't actually know if any of that is actually blocking, because as previously stated, I'm just here trying to unstick a PR without traction.

philiptaron · 2025-03-19T12:35:56Z

pkgs/by-name/ba/bazel_8/package.nix

+  makeWrapper,
+  writeTextFile,
+  writeScript,
+  substituteAll,


use replaceVars instead (#237216)

philiptaron · 2025-03-19T12:36:59Z

pkgs/by-name/ba/bazel_8/package.nix

+      (substituteAll {
+        src = ./strict_action_env.patch;
+        strictActionEnvPatch = defaultShellPath;
+      })


Suggested change

(substituteAll {

src = ./strict_action_env.patch;

strictActionEnvPatch = defaultShellPath;

})

(replaceVars ./strict_action_env.patch {

strictActionEnvPatch = defaultShellPath;

})

philiptaron · 2025-03-19T12:37:14Z

pkgs/by-name/ba/bazel_8/package.nix

+      (substituteAll {
+        src = ./bazel_rc.patch;
+        bazelSystemBazelRCPath = bazelRC;
+      })


Suggested change

(substituteAll {

src = ./bazel_rc.patch;

bazelSystemBazelRCPath = bazelRC;

})

(replaceVars ./bazel_rc.patch {

bazelSystemBazelRCPath = bazelRC;

})

philiptaron · 2025-03-19T12:38:53Z

pkgs/top-level/all-packages.nix

    bazel_self = bazel_7;
  };

+  bazel_8 = darwin.apple_sdk_11_0.callPackage ../by-name/ba/bazel_8/package.nix {


Is this needed now @NixOS/darwin-core? I think not...

philiptaron · 2025-03-19T12:40:47Z

pkgs/by-name/ba/bazel_8/package.nix

+  # Bootstrap an existing Bazel so we can vendor deps with vendor mode
+  bazelBootstrap = stdenv.mkDerivation rec {
+    name = "bazelBootstrap";
+
+    src =
+      {
+        x86_64-linux = fetchurl {
+          url = "https://github.com/bazelbuild/bazel/releases/download/${bootstrapVersion}/bazel_nojdk-${bootstrapVersion}-linux-x86_64";
+          hash = "sha256-KLejW8XcVQ3xD+kP9EGCRrODmHZwX7Sq3etdrVBNXHI=";
+        };
+        aarch64-linux = fetchurl {
+          url = "https://github.com/bazelbuild/bazel/releases/download/${bootstrapVersion}/bazel_nojdk-${bootstrapVersion}-linux-arm64";
+          hash = "sha256-+4GTKurt1KBkdyfC2hOLI31gBDOm1wTn91+kDX1di9M=";
+        };
+        x86_64-darwin = fetchurl {
+          url = "https://github.com/bazelbuild/bazel/releases/download/${bootstrapVersion}/bazel-${bootstrapVersion}-darwin-x86_64";
+          hash = "sha256-MsqLv4ZssUGQv7AZzhrEp9YbjLs/B3ETeXTo0OXN8es=";
+        };
+        aarch64-darwin = fetchurl {
+          url = "https://github.com/bazelbuild/bazel/releases/download/${bootstrapVersion}/bazel-${bootstrapVersion}-darwin-arm64";
+          hash = "sha256-0QrGSIVQxSEa7SAIT0Dft39jZ+IpsVoc8ocFeUHZMys=";
+        };
+      }
+      .${stdenv.hostPlatform.system};
+
+    nativeBuildInputs = defaultShellUtils;
+    buildInputs = [
+      stdenv.cc.cc
+    ] ++ lib.optional (!stdenv.hostPlatform.isDarwin) autoPatchelfHook;
+
+    dontUnpack = true;
+    dontPatch = true;
+    dontBuild = true;
+    dontStrip = true;
+    installPhase = ''
+      runHook preInstall
+
+      mkdir -p $out/bin
+      install -Dm755 $src $out/bin/bazel
+
+      runHook postInstall
+    '';
+
+    postFixup = ''
+      wrapProgram $out/bin/bazel \
+        --prefix PATH : ${lib.makeBinPath nativeBuildInputs}
+    '';
+
+    meta.sourceProvenance = with lib.sourceTypes; [ binaryNativeCode ];
+  };


Prefer to extract this to its own package:

bazelBootstrap = callPackage ./bootstrap.nix { inherit whatever };

philiptaron · 2025-03-19T13:07:33Z

pkgs/by-name/ba/bazel_8/package.nix

+        }
+
+        # Also build parser_deploy.jar with bootstrap bazel
+        # TODO: Turn into a proper patch


I'd really appreciate this. Sed is great for very targeted fixes, but this wide-scale rework is better as a patch, or a separate derivation.

philiptaron · 2025-03-19T13:08:40Z

pkgs/by-name/ba/bazel_8/package.nix

+        # reconstruct the now patched builtins_bzl.zip
+        pushd src/main/java/com/google/devtools/build/lib/bazel/rules/builtins_bzl_zip &>/dev/null
+          zip ../builtins_bzl.zip $(find builtins_bzl -type f) >/dev/null
+          rm -rf builtins_bzl


This made me think that making a separate derivation for builtins_bzl would work and clarify this mega derivation.

philiptaron · 2025-03-19T13:10:21Z

pkgs/by-name/ba/bazel_8/package.nix

+    # $out/bin/bazel-{version}-{os_arch} The binary _must_ exist with this
+    # naming if your project contains a .bazelversion file.
+    cp ./bazel_src/scripts/packages/bazel.sh $out/bin/bazel
+    versionned_bazel="$out/bin/bazel-${version}-${system}-${arch}"


Suggested change

versionned_bazel="$out/bin/bazel-${version}-${system}-${arch}"

versioned_bazel="$out/bin/bazel-${version}-${system}-${arch}"

minor typo

philiptaron · 2025-03-19T13:11:20Z

pkgs/by-name/ba/bazel_8/package.nix

+      # The string literal specifying the path to the bazel-rc file is sometimes
+      # stored non-contiguously in the binary due to gcc optimisations, which leads
+      # Nix to miss the hash when scanning for dependencies
+      echo "${bazelRC}" >> $out/nix-support/depends


That's pretty spooky!

philiptaron · 2025-03-19T13:12:30Z

pkgs/by-name/ba/bazel_8/package.nix

+    # TODO add some tests to cover basic functionality, and also tests for enableNixHacks=true (buildBazelPackage tests)
+    # tests = ...


🎶 wouldn't it be nice 🎶

boltzmannrain · 2025-03-24T11:21:55Z

Suggestions make sense to me 👍 But will need some time to implement. Also looks like 8.0.1->8.1.1 bump broke enableNixHacks patch and buildBazelPackage compatibility, will see how to fix and make it work with say bazelbuild/examples repo which'd also serve as a test. Maybe will need to implement vendor flow used in bazel_8 build for buildBazelPackage

hzeller · 2025-03-26T05:49:36Z

FWIW, I tested 8.1.1 from this PR on two of projects I maintain ( bant and verible), both of which are compatible with a wide range of bazel versions from 6.5.0 to 8.1.1.

One thing I noticed: the enableNixHacks patch does not apply cleanly, so this PR currently only builds in the configuration without. So currently using bazel_8 when building nix packages does not work as nix-package building is enabling that hack. So only compiled my projects locally with a fresh 8.1.1 build ( cd my_project; /nix/store/nyc8b16lkh434bqbmnb51ipwl392irkq-bazel-8.1.1/bin/bazel build -c opt ... )
Projects that use 8.1.x seem to implicitly upgrade rules_cc to some newer version, or start using rules_cc to run the compiler. In any case, that rules_cc is not new enough to contain the patch that makes them nix-compatible ( shebang: changed scripts shebang to better support systems like bsd and nixos bazelbuild/rules_cc#306 ) which is not in a released version yet.

So I can only use 8.1.1 from this PR on these projects if I force upgrading to a working rules_cc version by adding the following to my projects' MODULE.bazel

bazel_dep(name = "rules_cc", version = "0.1.1")
git_override(
    module_name = "rules_cc",
    commit = "d74915024017250e46d95e91a3defc34174effe0",  # get commit of rules_cc PR 306
    remote = "https://github.com/bazelbuild/rules_cc",
)

With that, both bant and verible build.

I wonder if in the 8.1.1 somewhere has its MODULE.bazel that loads some older rules_cc, and maybe we can patch it up to contain that override for it to work ?

hzeller · 2025-03-26T06:07:01Z

Uh, looking through the bazel 8.1.1 source code to see what rules_cc they use ... I only see references to somewhat ancient ones (also not sure with is implicitly used when bazel is run on a project)

bazel[release-8.1.1]$ rg rules_cc.*version
src/MODULE.tools:bazel_dep(name = "rules_cc", version = "0.0.17")
src/test/py/bazel/bazel_windows_test.py:            'bazel_dep(name = "rules_cc", version = "0.0.12")',
src/test/py/bazel/launcher_test.py:            'bazel_dep(name = "rules_cc", version = "0.0.12")',
src/test/py/bazel/testdata/runfiles_test/MODULE.bazel.mock:bazel_dep(name = "rules_cc", version = "0.0.9")
src/test/py/bazel/bazel_windows_cpp_test.py:            'bazel_dep(name = "rules_cc", version = "0.0.12")',
site/en/external/module.md:bazel_dep(name = "rules_cc", version = "0.0.1")
site/en/external/overview.md:bazel_dep(name = "rules_cc", version = "0.0.1")
MODULE.bazel:bazel_dep(name = "rules_cc", version = "0.0.17")

... not sure if my suggestion to upgrade rules_cc to something newer than 0.1.1 is feasible if it is so far off.

hzeller · 2025-03-30T22:16:50Z

... not sure if my suggestion to upgrade rules_cc to something newer than 0.1.1 is feasible if it is so far off.

I am convinced now this is not something we should work around in bazel 8. Bazel 8 actually just introduces a minimal rules_cc and that is fine. Other projects might require a newer rules_cc because they depend on other things that pull in a rules_cc that has not the fix yet. So I think nothing to block this PR, it is up to the projects compiling things with bazel 8+ to fix that.

So LGTM to me.

boltzmannrain · 2025-04-01T22:01:58Z

Marking as draft for now, will probably create a new one after refactoring and making tests work

IanMcShane · 2025-04-19T17:53:18Z

Super excited about this PR. Thanks for the effort!

boltzmannrain · 2025-04-20T17:15:42Z

Bazel 8.2.1 is much easier to bootstrap-build, so hope to open a new improved PR soon

boltzmannrain · 2025-04-22T21:12:00Z

Draft for 8.2.1 #400941 I will be refactoring and cleaning it up a fair bit but big-picture is settled for now

boltzmannrain · 2025-08-31T14:10:45Z

8.2.1 PR supersedes 8.1.1

boltzmannrain force-pushed the bazel_8_rc branch from ef3dddc to 86a0496 Compare December 6, 2024 18:53

github-actions bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Dec 6, 2024

boltzmannrain force-pushed the bazel_8_rc branch from 86a0496 to b0bfdb5 Compare December 6, 2024 19:00

This comment was marked as outdated.

Sign in to view

ofborg bot added the 8.has: package (new) This PR adds a new package label Dec 7, 2024

ofborg bot requested review from aherrmann, avdv, groodt, mboes, olebedev, uri-canva and ylecornec December 7, 2024 14:40

ofborg bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Dec 7, 2024

boltzmannrain force-pushed the bazel_8_rc branch from b0bfdb5 to efaf838 Compare December 8, 2024 22:01

boltzmannrain changed the title ~~bazel_8: init at 8.0.0rc7~~ bazel_8: init at 8.0.0rc8 Dec 8, 2024

This comment was marked as outdated.

Sign in to view

github-actions bot removed 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Dec 8, 2024

boltzmannrain force-pushed the bazel_8_rc branch from efaf838 to 960a941 Compare December 8, 2024 22:20

This comment was marked as outdated.

Sign in to view

boltzmannrain force-pushed the bazel_8_rc branch from 960a941 to 458f87c Compare December 8, 2024 22:41

This comment was marked as outdated.

Sign in to view

ofborg bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Dec 9, 2024

boltzmannrain force-pushed the bazel_8_rc branch 2 times, most recently from 2840557 to b6ecb11 Compare December 10, 2024 23:50

github-actions bot removed 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Dec 10, 2024

nix-owners bot requested review from infinisil and philiptaron January 19, 2025 13:26

infinisil reviewed Jan 20, 2025

View reviewed changes

philiptaron requested a review from a team January 23, 2025 13:53

seh reviewed Mar 5, 2025

View reviewed changes

boltzmannrain force-pushed the bazel_8_rc branch from d770743 to 4e41ee3 Compare March 5, 2025 22:47

github-actions bot removed the 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions label Mar 5, 2025

boltzmannrain changed the title ~~bazel_8: init at 8.0.1~~ bazel_8: init at 8.1.1 Mar 5, 2025

boltzmannrain force-pushed the bazel_8_rc branch from 4e41ee3 to a896868 Compare March 5, 2025 22:52

github-actions bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Mar 5, 2025

bazel_8: init at 8.1.1

fae10f5

philiptaron reviewed Mar 19, 2025

View reviewed changes

dabrowski-adam mentioned this pull request Apr 17, 2025

Update Request: Bazel 6.5.0 → v8.1.1 #398607

Closed

3 tasks

boltzmannrain mentioned this pull request Oct 31, 2025

bazel_7: fix bazelDeps build on darwin under sandbox=true #457328

Open

13 tasks

	versionned_bazel="$out/bin/bazel-${version}-${system}-${arch}"
	versioned_bazel="$out/bin/bazel-${version}-${system}-${arch}"

		# TODO add some tests to cover basic functionality, and also tests for enableNixHacks=true (buildBazelPackage tests)
		# tests = ...

Uh oh!

Conversation

boltzmannrain commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Choose a reason for hiding this comment

Uh oh!

timothyklim commented Mar 5, 2025

Uh oh!

seh commented Mar 5, 2025

Uh oh!

seh commented Mar 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

boltzmannrain commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timothyklim commented Mar 19, 2025

Uh oh!

philiptaron commented Mar 19, 2025

nixpkgs-review result

x86_64-darwin

Uh oh!

philiptaron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

boltzmannrain commented Mar 24, 2025

Uh oh!

hzeller commented Mar 26, 2025

Uh oh!

hzeller commented Mar 26, 2025

Uh oh!

hzeller commented Mar 30, 2025

Uh oh!

boltzmannrain commented Apr 1, 2025

Uh oh!

IanMcShane commented Apr 19, 2025

Uh oh!

boltzmannrain commented Apr 20, 2025

Uh oh!

boltzmannrain commented Apr 22, 2025

Uh oh!

boltzmannrain commented Aug 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

boltzmannrain commented Dec 6, 2024 •

edited

Loading

boltzmannrain commented Mar 6, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-darwin`