Download a pre-compiled bootstrap from CI

[jyn514]

Currently, we unconditionally recompile bootstrap on all builds, like libstd. In most cases, this is unnecessary: bootstrap is rarely changed and it's confusing that we have to build a whole rust program before processing x.py --help. Additionally, it means that when we add the new shell script entrypoints, we'd have to replicate all of bootstrap.py's logic for finding config.toml and the build directory. If we can download bootstrap through rustup, it will greatly simplify all that logic.

Most of this work has already been done in #98483. The work left is to actually download the generated artifact in bootstrap.py.

cc @dvtkrlbs

Mentoring instructions:

1. Add a new download_or_build_bootstrap function to src/bootstrap/bootstrap.py.
2. If there are changes to src/bootstrap (you can find these with git diff-index --quiet HEAD -- src/bootstrap), then always rebuild it from source.
3. Otherwise, download it from ci-artifacts.rust-lang.org, as configured by stage0.json - you may have to modify the Stage0Toolchain python object to load this configuration.

Helps with #94829.

[dvtkrlbs]

@rustbot claim

[est31]

bootstrap is rarely changed

I'm not sure I agree. According to git log --pretty=format:"%ad" --author "[email protected]" src/bootstrap/ | rg 2022 | wc -l there have been 73 PRs merged this year that changed something in bootstrap. If we assume 211 days since Jan 1, there has been a change every 2.9 days. Compare that to 16 updates of src/llvm-project in 2022 (obtained with a similar command, but bors commits don't show up in llvm-project).

But it's still bearable I think. Definitely worth the cost of hosting a few binaries.

[jyn514]

@est31 I'd guess probably a good half of those are related to #94829. We should be seeing less churn once that gets closed out.

A fair number of the rest are related to #44293, which has also been fixed and won't have more PRs in the future.

Bootstrap is definitely changing more than LLVM, but I don't know that it's a great metric to compare to; I think bootstrap is a failiy small download compared to everything else.

[Mark-Simulacrum]

It's also true that we could tune the divide here over time - for example, splitting bootstrap into configuration+ flag parsing and the inner bits which actually run things.

Ultimately, the primary goal with downloading it is to reduce the amount of non-Rust code we maintain, and we need a pretty thin shim to parse flags, config, and download bootstrap compiler to proceed from there.

But yes, we should make sure the binary is not unexpectedly large on download - hopefully just a few megabytes.

[jyn514]

@dvtkrlbs 👋 it's been a while - are you still planning to work on this?

[est31]

FTR if you run the command from above then you get 13 PRs that changed bootstrap in 2023, which gives a 11 days per PR average rate, which is close to the LLVM change rate quoted above. And with serious build time increases proposed, like the 20%+ increase proposed in #108083, this makes more sense than it used to IMO.

[dvtkrlbs]

@dvtkrlbs 👋 it's been a while - are you still planning to work on this?

sadly no I am removing myself from the issue.

[est31]

Another proposed PR that causes significant build time increases: #110369

[mental32]

@rustbot claim

[mental32]

unclaiming as i have zero bandwith for this and some time has passed. @jyn514 sorry dude!

[dvtkrlbs]

@rustbot claim

[dvtkrlbs]

I am reclaiming this issue since I have time to work on it. One question @jyn514 what is the directory structure for the artifacts server?

[dvtkrlbs]

Another side note is that it seems we are not generating checksums for the bootstrap artifacts

[jyn514]

Another side note is that it seems we are not generating checksums for the bootstrap artifacts

yes, this is known and expected, we don't have checksums for download-rustc artifacts either. I ripped out the python code that made verification optional a while back, feel free to add it back.

what is the directory structure for the artifacts server?

see _download_component_helper - I think you should be able to call it with filename=bootstrap and have it work? if not, take a look at dist.rs to see what we named the tarball all those months ago.

[dvtkrlbs]

Hmm it is bootstrap but the problem is the dist server does not seem to have any bootstrap artifacts are they not enabled to be pushed to the static server. Or should I've been looking at the artifacts_server instead of dist_server?

[jyn514]

ah yeah that's probably it - ci-artifacts.rust-lang.org is different than static.rust-lang.org. you can add debugging to src/bootstrap/download.rs to see what it's doing, it uses ci-artifacts for download-rustc.

[dvtkrlbs]

i am going to push my local changes which downloads the bootstrap correctly but i cant compreehend the unzip part of the python code which is has couple level of indirections. right now the bootstrap binary ends up in "build\x86_64-pc-windows-msvc\stage0\nightly-x86_64-pc-windows-msvc\bootstrap\bootstrap\bin\bootstrap.exe" and i am not sure if i am supposed to move this to build/bootstrap/debug or make the bootstarp bin dir function also handle git changes

[dvtkrlbs]

i think the correct way to handle this is to update bootstrap_binary and also check the diff-index and based on that going into the cached binary path but i first want to make sure my approach is correct. pushed latest changes @jyn514

[dvtkrlbs]

Sorry for the lack of progress on this. I've been quite busy for a while and couldn't find any time to work on this I am closing this PR in case someone wants to continue feel free to base from my changes.