Skip to content

Conversation

@trofi
Copy link
Contributor

@trofi trofi commented Jan 10, 2023

Autoreconf was initially added in e44dd84 ("libtool2: macOS 11 support") for configure patches. The patches were removed since 2.4.7 release. Let's drop these dependencies to make boostrap tree leaner.

The diff of bootstrap tree before and after the change:

$ nix-store --query --graph $(nix-instantiate -A stdenv) |
    fgrep ' -> ' | awk '{print $3}' | sort -u |
    sed 's/"[0-9a-z]\{32\}-/"/g' | sort > before

$ nix-store --query --graph $(nix-instantiate -A stdenv) |
    fgrep ' -> ' | awk '{print $3}' | sort -u |
    sed 's/"[0-9a-z]\{32\}-/"/g' | sort > after

$ diff -U0 before after

--- before
+++ after
@@ -64,2 +63,0 @@
-"help2man-1.49.2.drv"
-"help2man-1.49.2.drv"
@@ -77 +74,0 @@
-"libxcrypt-4.4.33.drv"
@@ -88,4 +84,0 @@
-"perl-5.36.0.drv"
-"perl-5.36.0.drv"
-"perl5.36.0-gettext-1.07.drv"
-"perl5.36.0-gettext-1.07.drv"

This removes 2 of 3 builds of perl and help2man dependencies.

Description of changes
Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch. 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Jan 11, 2023
@ghost ghost mentioned this pull request Jan 13, 2023
4 tasks
Autoreconf was initially added in e44dd84 ("libtool2: macOS 11
support") for configure patches. The patches were removed since 2.4.7
release. Let's drop these dependencies to make boostrap tree leaner.

The diff of bootstrap tree before and after the change:

    $ nix-store --query --graph $(nix-instantiate -A stdenv) |
        fgrep ' -> ' | awk '{print $3}' | sort -u |
        sed 's/"[0-9a-z]\{32\}-/"/g' | sort > before

    $ nix-store --query --graph $(nix-instantiate -A stdenv) |
        fgrep ' -> ' | awk '{print $3}' | sort -u |
        sed 's/"[0-9a-z]\{32\}-/"/g' | sort > after

    $ diff -U0 before after

    --- before
    +++ after
    @@ -64,2 +63,0 @@
    -"help2man-1.49.2.drv"
    -"help2man-1.49.2.drv"
    @@ -77 +74,0 @@
    -"libxcrypt-4.4.33.drv"
    @@ -88,4 +84,0 @@
    -"perl-5.36.0.drv"
    -"perl-5.36.0.drv"
    -"perl5.36.0-gettext-1.07.drv"
    -"perl5.36.0-gettext-1.07.drv"

This removes 2 of 3 builds of `perl` and `help2man` dependencies.

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
@trofi trofi force-pushed the libtool-without-perl branch from 596f56b to 6becbd3 Compare January 13, 2023 22:20
@Artturin Artturin merged commit 5e2d7ff into NixOS:staging Jan 14, 2023
@trofi trofi deleted the libtool-without-perl branch January 14, 2023 07:57
@Cynerd
Copy link
Contributor

Cynerd commented Jan 16, 2023

This seem to break build on aarch64 for me.

help2man: can't get `--help' info from libtoolize
Try `--no-discard-stderr' if option outputs to stderr
WARNING: 'help2man' is missing on your system.
         You should only need it if you modified a dependency of a man page.
         You may want to install the GNU Help2man package:
         <https://www.gnu.org/software/help2man/>

@amjoseph-nixpkgs is that something you are tackling as part of gcc11 for aarch64?

@Artturin
Copy link
Member

This seem to break build on aarch64 for me.

help2man: can't get `--help' info from libtoolize
Try `--no-discard-stderr' if option outputs to stderr
WARNING: 'help2man' is missing on your system.
         You should only need it if you modified a dependency of a man page.
         You may want to install the GNU Help2man package:
         <https://www.gnu.org/software/help2man/>

which build is that from? libtool builds for me on e09153d on aarch64-linux

@Cynerd
Copy link
Contributor

Cynerd commented Jan 16, 2023

The latest nixos-unstable-small (c85d086). I should add that it is not a native build but cross build.

@Artturin
Copy link
Member

this PR isn't even in master yet https://nixpk.gs/pr-tracker.html?pr=210118 so your issue is from some other commit

@Cynerd
Copy link
Contributor

Cynerd commented Jan 16, 2023

Oh.. sorry. You are right. The build process still includes help2man in my version. I jumped at this because I saw it being removed here. I have to investigate more than to know why it reports that it is not available in my case.

@trofi
Copy link
Contributor Author

trofi commented Jan 16, 2023

Do you apply local patches to libtool by chance? That could trigger manual rebuilds. If you want to do it locally you might need to pull in extra depends to rebuild docs (or ad a similar touch hack to avoid the extra depends).

@Cynerd
Copy link
Contributor

Cynerd commented Jan 17, 2023

Do you apply local patches to libtool by chance?

That would be my idea as well, but no. I am not applying any patches.

I tried to bisect the commit that might have broke it, but I ended up with a working build after bisect even on nixos-unstable-small. There is something smelly in libtool build.

ConnorBaker pushed a commit to nixos-cuda/cuda-legacy that referenced this pull request Apr 14, 2025
 #### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like NixOS/nixpkgs#132343

 #### Incorporates

- NixOS/nixpkgs#210004
- NixOS/nixpkgs#36948 (unreverted)
- NixOS/nixpkgs#210325
- NixOS/nixpkgs#210118
- NixOS/nixpkgs#210132
- NixOS/nixpkgs#210109
- NixOS/nixpkgs#213909
- NixOS/nixpkgs#216136
- NixOS/nixpkgs#216237
- NixOS/nixpkgs#210019
- NixOS/nixpkgs#216232
- NixOS/nixpkgs#216016
- NixOS/nixpkgs#217977
- NixOS/nixpkgs#217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @Ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: NixOS/nixpkgs#210004
[versioned directory]: NixOS/nixpkgs#209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253
[mentioned]: NixOS/nixpkgs#210112 (comment)
[crisis]: NixOS/nixpkgs#108305
[foreign]: NixOS/nixpkgs#170857 (comment)
[static lib{mpfr,mpc,gmp,isl}.a hack]: https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants