minimal-bootstrap: point bootstrap sources to tarballs.nixos.org by emilytrau · Pull Request #232576 · NixOS/nixpkgs

emilytrau · 2023-05-18T08:36:34Z

Description of changes

The purpose of this PR is to move the initial source tarball to an official location as discussed in #227914 (comment). To facilitate I have exposed the packaging steps as the minimal-bootstrap-sources package.

Related #227914

Things done

pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/bootstrap-sources.nix

pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/make-bootstrap-sources.nix

lovesegfault · 2023-05-18T11:23:21Z

Hi @emilytrau!

Thrilled to see this work progress :)

Could you take a look at the format used here to request tarball uploads? #183487

If you could make a comment in that shape, it'd help me a lot in uploading this quickly

Thanks!

emilytrau · 2023-05-18T11:27:31Z

@lovesegfault thanks so much!

lovesegfault · 2023-05-18T11:36:08Z

(Might take a couple of days, I'm travelling and didn't bring my upload keys)

ghost · 2023-06-08T20:27:57Z

(I am back and close to conquering my backlog)

Quick glance LGTM; will do a closer review this weekend if this hasn't been merged by then.

Ericson2314 · 2023-06-10T15:53:20Z

Is this undrafted because the tarballs are uploaded now?

ghost

This has to be done in two separately-merged PRs, with a Hydra build occurring after the first but before the second. See below. Yeah, it's annoying, I know, I went through it four or five times.

ghost · 2023-06-15T03:44:50Z

pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/bootstrap-sources.nix

+
+  #
+  # Files came from this Hydra build:
+  #
+  #   https://hydra.nixos.org/build/<placeholder>
+  #
+  # Which used nixpkgs revision <placeholder>
+  # to instantiate:
+  #
+  #   /nix/store/<placeholder>.drv
+  #
+  # and then built:
+  #
+  #   /nix/store/<placeholder>
+  #


Suggested change

#

# Files came from this Hydra build:

#

# https://hydra.nixos.org/build/<placeholder>

#

# Which used nixpkgs revision <placeholder>

# to instantiate:

#

# /nix/store/<placeholder>.drv

#

# and then built:

#

# /nix/store/<placeholder>

#

ghost · 2023-06-15T03:46:55Z

pkgs/top-level/all-packages.nix

    inherit (stdenv) buildPlatform hostPlatform;
    inherit lib config;
  });
+  minimal-bootstrap-sources = callPackage ../os-specific/linux/minimal-bootstrap/stage0-posix/make-bootstrap-sources.nix { };


There are no Hydra builds of minimal-bootstrap-sources yet.

In order to get a Hydra build you need to:

merge a PR that has the line above

wait for Hydra to build it

then submit a separate PR which references that Hydra build.

The second PR is the one that should use the template @lovesegfault referenced.

Also, the provenance of builds on tarballs.nixos.org is a big deal. You should delete the fetchurl of https://github.com/emilytrau/bootstrap-tools-nar-mirror/ so it isn't anywhere in the nixpkgs repo at the commit from which Hydra builds the narball. Otherwise it's going to raise circularity suspicions.

ghost · 2023-06-15T03:57:31Z

pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/bootstrap-sources.nix

  src = import <nix/fetchurl.nix> {
    inherit name;


Suggested change

src = import <nix/fetchurl.nix> {

inherit name;

ghost · 2023-06-15T03:58:48Z

pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/bootstrap-sources.nix

+    url = "https://github.com/emilytrau/bootstrap-tools-nar-mirror/releases/download/2023-05-18/${name}.nar.xz";
+    hash = "sha256-FpMp7z+B3cR3LkQ+PooH/b1/NlxH8NHVJNWifaPWt4U=";
    unpack = true;
  };
 }


Suggested change

url = "https://github.com/emilytrau/bootstrap-tools-nar-mirror/releases/download/2023-05-18/${name}.nar.xz";

hash = "sha256-FpMp7z+B3cR3LkQ+PooH/b1/NlxH8NHVJNWifaPWt4U=";

unpack = true;

};

}

Please delete this, wait for Hydra to do the build, then re-add it using the tarballs.nixos.org url. That way there won't be any doubts raised about the commit at which Hydra does the build having access to this narball of questionable provenance.

ghost · 2023-06-15T03:59:56Z

pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/make-bootstrap-sources.nix

 # To build:
 #
-#   nix-build pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/make-bootstrap-sources.nix
+#   nix-build . -A minimal-bootstrap-sources


Suggested change

# nix-build . -A minimal-bootstrap-sources

# nix-build '<nixpkgs>' -A minimal-bootstrap-sources

ghost · 2023-06-15T04:05:48Z

I have to say it is kind of a bummer that we still end up embedding a "magic hash" of some Hydra build. I was kinda hoping the minimal-bootstrap would let us avoid doing that.

Of course, it's less-bad since that Hydra build contains no binaries, but still...

emilytrau · 2023-06-15T04:40:40Z

@amjoseph-nixpkgs iirc builtins.fetchGit isn't available in restricted evaluation environments even with a hash declared.

Also I suppose now that I think of it again, the purpose of uploading this source tarball to tarballs.nixos.org is really just to establish the process in which others can update it with stage0-posix releases, as opposed to having it on my personal GitHub repo. As it's all source code it wouldn't differ in terms of security than fetching a tarball for any other package from any 3rd party source, and the "clean room" benefits of building in hydra maybe isn't integral?

Maybe persuading upstream to publish release tarballs in nar or shar formats would also solve the issue of maintenance.

emilytrau · 2023-06-15T05:09:53Z

@amjoseph-nixpkgs another suggestion might be to move the tarball hosting repo to nix-community. The process of generating a nar from a stage0-posix release is almost already automated and could just need someone to open a PR bumping the version whenever needed. It would definitely be easier than going through this tarballs.nixos.org process every time.

ghost · 2023-06-15T05:58:28Z

Ah yes, now I remember why my scheme is doomed to fail:

It isn't documented in the manual, but builder="builtins:fetchurl" is the "cheat code" that lets you skip the restricted-mode allowed-uris check. If you use builtins.fetchurl or builtins.fetchGit or builtins.fetchTarball you don't get to skip that check.

There is no builder="builtins:fetchgit" or builder="builtins:fetchtarball".

In fact, there is no "cheat-code" that lets you access any of the libarchive unpacking functionality that is built into Nix (except for xz-decompression).

That became a problem for NixOS's channels, so another "cheat-code" was added for that use case: builder="builtins:unpack-channel". But it barfs if the tarball contains more than one file.

So yeah, we have to put a nar somewhere.

Anyways, the split-it-into-two-PRs part of my review still stands; if you could please address that I'll approve this, we can merge it, and start waiting for Hydra to do the build.

ghost · 2023-06-15T06:09:49Z

@amjoseph-nixpkgs another suggestion might be to move the tarball hosting repo to nix-community.

Or have somebody run a streaming HTTP GET tarball-to-narball converter and point ./pkgs/build-support/fetchurl/boot.nix at that.

curl -o stage0-posix.nar \
  http://nixos.org/tar2nar/https://github.com/oriansj/stage0-posix/archive/3189b5f325b7ef8b88e3edec7c1cde4fce73c76c.tar

Ericson2314 · 2023-06-24T18:09:53Z

I am not knowledgeable about this Hydra policy tarball stuff, and @amjoseph-nixpkgs is, so I defer to him.

Ericson2314 · 2023-06-24T18:10:39Z

#238357 does make sense to me though. Does it make sense to you @emilytrau as a replacement?

emilytrau · 2023-06-24T23:08:16Z

#238357 makes a lot of sense to me as well :)

This commit adjusts NixOS#232576 to break the fetchurl<->minimal-bootstrap-sources dependency cycle without needing an upload to tarballs.nixos.org. It does this by appending the low-level FOD attributes onto the `runCommand` derivation. https://nixos.org/manual/nix/unstable/language/advanced-attributes.html#adv-attr-outputHash

emilytrau assigned lovesegfault May 18, 2023

emilytrau mentioned this pull request May 18, 2023

stdenv: begin work on 256b bootstrap i.e. Trusting Trust #227914

Merged

12 tasks

ofborg bot added the 8.has: package (new) This PR adds a new package label May 18, 2023

ofborg bot requested review from 06kellyjac, Artturin and Ericson2314 May 18, 2023 08:54

06kellyjac reviewed May 18, 2023

View reviewed changes

pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/bootstrap-sources.nix Outdated Show resolved Hide resolved

pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/make-bootstrap-sources.nix Outdated Show resolved Hide resolved

minimal-bootstrap: create top-level attr for bootstrap sources

cede8c0

emilytrau force-pushed the minimal-bootstrap-sources branch from 01a5127 to cede8c0 Compare May 18, 2023 11:27

lovesegfault requested review from a user and trofi May 18, 2023 11:32

ofborg bot requested a review from 06kellyjac May 18, 2023 12:22

trofi removed their request for review May 18, 2023 12:44

ghost suggested changes Jun 15, 2023

View reviewed changes

This comment was marked as outdated.

Sign in to view

ghost mentioned this pull request Jun 15, 2023

src/libexpr/eval.hh: fix typo NixOS/nix#8522

Merged

8 tasks

ghost mentioned this pull request Jun 15, 2023

minimal-bootstrap: use builtins.fetchGit #237863

Closed

12 tasks

ghost mentioned this pull request Jun 18, 2023

minimal-bootstrap: make sources a non-tarballs.nixos.org FOD #238357

Merged

12 tasks

emilytrau closed this Jun 24, 2023

emilytrau deleted the minimal-bootstrap-sources branch June 24, 2023 23:11

	# nix-build . -A minimal-bootstrap-sources
	# nix-build '<nixpkgs>' -A minimal-bootstrap-sources

Uh oh!

Conversation

emilytrau commented May 18, 2023

Description of changes

Things done

Uh oh!

Uh oh!

Uh oh!

lovesegfault commented May 18, 2023

Uh oh!

emilytrau commented May 18, 2023

Uh oh!

lovesegfault commented May 18, 2023

Uh oh!

ghost commented Jun 8, 2023

Uh oh!

Ericson2314 commented Jun 10, 2023

Uh oh!

ghost left a comment • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost Jun 15, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jun 15, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jun 15, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jun 15, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jun 15, 2023

Choose a reason for hiding this comment

Uh oh!

ghost commented Jun 15, 2023 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

emilytrau commented Jun 15, 2023

Uh oh!

This comment was marked as outdated.

emilytrau commented Jun 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Jun 15, 2023 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Jun 15, 2023

Uh oh!

Ericson2314 commented Jun 24, 2023

Uh oh!

Ericson2314 commented Jun 24, 2023

Uh oh!

emilytrau commented Jun 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ghost left a comment •

edited by ghost

Loading

ghost commented Jun 15, 2023 •

edited by ghost

Loading

emilytrau commented Jun 15, 2023 •

edited

Loading

ghost commented Jun 15, 2023 •

edited by ghost

Loading