Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Making alt build available together with Nightly #45334

Closed
SimonSapin opened this issue Oct 17, 2017 · 4 comments
Closed

Making alt build available together with Nightly #45334

SimonSapin opened this issue Oct 17, 2017 · 4 comments
Labels
T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@SimonSapin
Copy link
Contributor

SimonSapin commented Oct 17, 2017 •
edited
Loading

In https://internals.rust-lang.org/t/updates-on-rusts-ci-uploads/6062 I found out that the rust-lang-ci S3 bucket, which Servo relies on, is ephemeral. Only nigthlies are kept around long-term. I’ll switch Servo to download from http://static.rust-lang.org/dist/ where possible, but there’s one thing missing: tarballs for rustc "alt" builds. I’d like to have these somewhere where they’ll stay available.

I assume that the reason for removing old files from rust-lang-ci / rust-lang-ci2 is the cost in storage space for many build artifacts for every single merged PR. So this proposal limits the amount of files to keep:

  • To one set per Nightly version
  • To just one rustc tarball per supported platform.

First, we could limit the amount of files uploaded to rust-lang-ci2/rustc-builds-alt in the first place. Based on running aws s3 ls --no-sign-request, CI produces:

  • cargo-nightly-$HOST.tar.$Z
  • rls-nightly-$HOST.tar.$Z
  • rust-analysis-nightly-$HOST.tar.$Z
  • rustc-nightly-$HOST.tar.$Z
  • rustc-nightly-src.tar.$Z
  • rust-docs-nightly-$HOST.tar.$Z
  • rust-nightly-$HOST.exe
  • rust-nightly-$HOST.msi
  • rust-nightly-$HOST.pkg
  • rust-nightly-$HOST.tar.$Z
  • rust-src-nightly.tar.$Z
  • rust-std-nightly-$HOST.tar.$Z

… for each combination of $Z in gz or xz, and $HOST in x86_64-apple-darwin, x86_64-pc-windows-msvc, and x86_64-unknown-linux-gnu.

Everything other than rustc-nightly-$HOST.tar.$Z is identical in rustc-builds and rustc-builds-alt. So it could be dropped or not uploaded in the first place or not not built in the first place. This probably involves making src/ci/docker/dist-x86_64-linux/Dockerfile do things more differently when the DEPLOY_ALT env variable is set.

(Perhaps gz can be dropped too? Old rustup versions that don’t support xz don’t support fetching alt builds either.)

Separately, https://github.com/rust-lang/rust-central-station/blob/master/promote-release/src/main.rs (at least) needs to be modified to copy/upload these extra tarballs. I don’t know if it’s preferable to do this when self.release == "nightly" or in a separate cron job with self.release == "nightly-alt".

Ultimately this will also be a step toward rust-lang/rustup#1099, but Servo can already use this in the meantime.

@aturon, @alexcrichton, how does this all sound?
SimonSapin added a commit to servo/servo that referenced this issue Oct 17, 2017
@SimonSapin
Download rustc from long-term URLs when possible.
8298111 
See https://internals.rust-lang.org/t/updates-on-rusts-ci-uploads/6062

Alt builds are still only available at ephemeral URLs for now:
rust-lang/rust#45334
@SimonSapin SimonSapin mentioned this issue Oct 17, 2017
Closed
@SimonSapin
Copy link
Contributor Author

Everything other than rustc-nightly-$HOST.tar.$Z is identical in rustc-builds and rustc-builds-alt.

Hmm maybe rust-nightly-$HOST.* is not, but neither rustup nor Servo uses those IIRC.

@aturon aturon added I-nominated T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Oct 17, 2017
@aturon
Copy link
Member

aturon commented Oct 17, 2017

cc @rust-lang/infra, nominating for discussion in tomorrow's meeting.

SimonSapin added a commit to SimonSapin/rust-central-station that referenced this issue Oct 28, 2017
@SimonSapin
Promote "alternative" rustc builds to Nightly archives
20bb517 
Nightly builds are archived under `https://static.rust-lang.org/dist/{date}/`.
This adds a new `alt/` subdirectory containing:

* `rustc-nightly-x86_64-apple-darwin.tar.gz`
* `rustc-nightly-x86_64-unknown-linux-gnu.tar.gz`
* `rustc-nightly-x86_64-pc-windows-msvc.tar.gz`

copied from `s3://rust-lang-ci2/rustc-builds-alt/`.
These are alternative builds of rustc with LLVM assertions disabled,
which reduces testing coverage but improves compiler performance.

This adds these resources to those that are considered “stable and public”
per https://internals.rust-lang.org/t/public-stable-rust-services/6072.
For the latest Nightly they take up 143 MiB.
At current S3 pricing, this increases the cost of storing 10 years worth
of Nightlies by 14 USD per month.

This fixes rust-lang/rust#45334
and will enable Servo’s build system to stop relying
on private and ephemeral resources.
@SimonSapin SimonSapin mentioned this issue Oct 28, 2017
Closed
@aidanhs
Copy link
Member

aidanhs commented Oct 31, 2017

Seems like progress is being made in https://internals.rust-lang.org/t/disabling-llvm-assertions-in-nightly-builds/5388/14

@SimonSapin
Copy link
Contributor Author

#45810 makes this obsolete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Promote "alternative" rustc builds to Nightly archives SimonSapin/rust-central-station
3 participants
@SimonSapin @aturon @aidanhs