Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement release process for rustup #84

Draft
wants to merge 28 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
00e1a90
Fix error message when parsing actions
jdno May 20, 2024
0a64d25
Create action for rustup
jdno May 20, 2024
8d4a447
Implement release process for rustup
jdno May 22, 2024
d40ff45
Extract channel check for Rustup into function
jdno Jun 27, 2024
b803170
Get latest commit from Rustup's stable branch
jdno Jun 27, 2024
da3164d
Fetch next Rustup version from GitHub
jdno Jun 27, 2024
49980a3
Download Rustup artifacts for a given commit
jdno Jun 27, 2024
c4cff77
Pass Rustup version to archive and manifest
jdno Jun 27, 2024
7ccff04
Update documentation for Rustup release process
jdno Jun 27, 2024
ede844c
Refactor run script to execute different local releases
jdno Sep 24, 2024
1e516af
Support other platforms than Linux
jdno Sep 25, 2024
b1dd2a2
Download Rustup files from CDN
jdno Oct 1, 2024
b843e51
Build promote-release inside the container if necessary
jdno Oct 1, 2024
f5c3c5e
Get download path from config
jdno Oct 1, 2024
a947033
Fix upload directory
jdno Oct 1, 2024
883b5ed
Get version from user-provided commit
jdno Oct 1, 2024
ad86e13
Don't require GitHub credentials to fetch Rustup metadata
jdno Oct 1, 2024
694e3a4
Create rustup-builds bucket in local environment
jdno Oct 1, 2024
8132d69
Fix upload destination for Rustup archives
jdno Oct 1, 2024
89f7324
Fix upload destination for stable Rustup release
jdno Oct 1, 2024
f82d531
Fix upload destination for Rustup manifest
jdno Oct 1, 2024
9b7dba8
Configure environment to run Rustup releases locally
jdno Oct 1, 2024
5aad90d
Set up CI workflows for refactored actions
jdno Oct 1, 2024
a52f975
Fix required jobs for deployments
jdno Oct 1, 2024
839b35d
Merge branch 'master' into promote-rustup
jdno Oct 2, 2024
ad0c357
Fix unnecessary borrow warnings
jdno Oct 3, 2024
fd3f06d
Cut beta releases for Rustup from the stable branch
jdno Oct 3, 2024
a3a0d89
Allow Rustup version to be overridden in tests
jdno Oct 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Implement release process for rustup
jdno committed May 23, 2024
commit 8d4a4476cbeb9ab6b3153df55738921fa2a9b83d
2 changes: 1 addition & 1 deletion src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ impl Context {
match self.config.action {
config::Action::PromoteRelease => self.do_release()?,
config::Action::PromoteBranches => self.do_branching()?,
config::Action::PromoteRustup => self.do_rustup()?,
config::Action::PromoteRustup => self.promote_rustup()?,
}
Ok(())
}
Expand Down
139 changes: 137 additions & 2 deletions src/rustup.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,142 @@
use crate::Context;
use std::fs;
use std::path::{Path, PathBuf};

use anyhow::{anyhow, Error};

use crate::config::Channel;
use crate::{run, Context};

impl Context {
pub fn do_rustup(&mut self) -> anyhow::Result<()> {
/// Promote a `rustup` release
///
/// The [release process] for `rustup` involves copying existing artifacts from one S3 bucket to
/// another, updating the manifest, and archiving the artifacts for long-term storage.
///
/// `rustup` uses different branches to manage releases. Whenever a commit is pushed to the
/// `stable` branch in [rust-lang/rustup], GitHub Actions workflows build release artifacts and
/// copy them into `s3://dev-static-rust-lang-org/rustup/dist/`.
///
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not great, I'd prefer if we tweak rustup's CI to upload them to a location like s3://rustup-artifacts/${commit}, check what is the latest commit hash on the stable branch, and download from that location. Every build overriding the previous build feels iffy.

Copy link
Member Author

@jdno jdno Jun 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bucket has been created here and a PR is open here to upload artifacts to s3://rustup-builds/builds/${commit}/.

/// When a new release is done and this method is invoked, it downloads the artifacts from that
/// bucket (which must always be set as the `DOWNLOAD_BUCKET` variable). A copy of the artifacts
/// is archived in `s3://${UPLOAD_BUCKET}/rustup/archive/${version}/`, where `version` is passed
/// to this program as a command-line argument. `UPLOAD_BUCKET` can either be the `dev-static`
/// or the `static` bucket.
///
/// If the release is for the `stable` channel, the artifacts are also copied to the `dist/`
/// path in the `UPLOAD_BUCKET` bucket. The `dist/` path is used by the `rustup` installer to
/// download the latest release.
///
/// Then, the `release-stable.toml` manifest is updated with the new version and copied to
/// `s3://${UPLOAD_BUCKET}/rustup/release-stable.toml`.
///
/// [release process]: https://rust-lang.github.io/rustup/dev-guide/release-process.html
/// [rust-lang/rustup]: https://github.com/rust-lang/rustup
pub fn promote_rustup(&mut self) -> anyhow::Result<()> {
println!("Checking channel...");
if self.config.channel != Channel::Stable && self.config.channel != Channel::Beta {
return Err(anyhow!(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels really weird, since the current environments for rustup are dev-static and prod.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we making the channel be the condition here? Promote release already has dev static and static, as two separate environments - should be able to ignore channels I'd expect. (In a manner similar to promote branches ignoring them iirc)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without channels, how would you distinguish between beta and stable releases? Only be setting the respective environment variables (e.g. UPLOAD_BUCKET)?

"promoting rustup is only supported for the stable and beta channels"
));
}

// Download the rustup artifacts from S3
println!("Downloading artifacts from dev-static...");
let dist_dir = self.download_rustup_artifacts()?;

// Archive the artifacts
println!("Archiving artifacts...");
self.archive_rustup_artifacts(&dist_dir)?;

if self.config.channel == Channel::Stable {
// Promote the artifacts to the release bucket
println!("Promoting artifacts to dist/...");
self.promote_rustup_artifacts(&dist_dir)?;
}

// Update the release number
println!("Updating version and manifest...");
self.update_rustup_release()?;

Ok(())
}

fn download_rustup_artifacts(&mut self) -> Result<PathBuf, Error> {
let dl = self.dl_dir().join("dist");
// Remove the directory if it exists, otherwise just ignore.
let _ = fs::remove_dir_all(&dl);
fs::create_dir_all(&dl)?;

run(self
.aws_s3()
.arg("cp")
.arg("--recursive")
.arg("--only-show-errors")
.arg(&self.s3_artifacts_url("dist/"))
.arg(format!("{}/", dl.display())))?;

Ok(dl)
}

fn archive_rustup_artifacts(&mut self, dist_dir: &Path) -> Result<(), Error> {
let version = self
.current_version
.as_ref()
.ok_or_else(|| anyhow!("failed to get current version for rustup release"))?;

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work, since self.current_version is None when this runs. Apparently, the version is set much later in the normal release process.

The easiest workaround might be setting a new PROMOTE_RELEASE_RUSTUP_VERSION environment variable, but that would need to be updated manually before running the CodeBuild project. That feels risky, since it's very easy to forget this step and right now we would just override existing artifacts. 😬

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should figure out how to persist the intended version with the artifacts. That should be possible similarly to how rust has a version containing file checked in, and we can upload that directly to the S3 bucket in rustup's existing "CI".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Mark that the best approach here is for rustup to store the version number in a file in the repository, so that they can update it on their own.

In general, regarding arguments, I don't see any problem with sticking with environment variables. We never invoke the CodeBuild job directly, we always go through https://github.com/rust-lang/simpleinfra/blob/master/release-scripts/promote-release.py, which has a CLI parser and then sets the environment variables.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An early step in our release process is a version bump in Cargo.toml(e.g. rust-lang/rustup@cfca13c). Is that useful for this particular purpose?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version is now read from the Cargo.toml in the latest commit on the stable branch.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@djc Now that we're at it, maybe it makes sense to use workspace-wide version numbers? I don't see how rustup-download can be at a different version from rustup itself.

Copy link
Member

@rami3l rami3l Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdno FYI: rust-lang/rustup#4041 rust-lang/rustup#4061 now uses a workspace-wide version number.

let path = format!("archive/{}/", version);

self.upload_rustup_artifacts(dist_dir, &path)
}

fn promote_rustup_artifacts(&mut self, dist_dir: &Path) -> Result<(), Error> {
let release_bucket_url = format!(
"s3://{}/{}/{}",
self.config.upload_bucket,
self.config.download_dir,
dist_dir.display(),
);

run(self
.aws_s3()
.arg("cp")
.arg("--recursive")
.arg("--only-show-errors")
.arg(format!("{}/", dist_dir.display()))
.arg(&release_bucket_url))
}

fn upload_rustup_artifacts(&mut self, dist_dir: &Path, target_path: &str) -> Result<(), Error> {
run(self
.aws_s3()
.arg("cp")
.arg("--recursive")
.arg("--only-show-errors")
.arg(format!("{}/", dist_dir.display()))
.arg(&self.s3_artifacts_url(target_path)))
}

fn update_rustup_release(&mut self) -> Result<(), Error> {
let version = self
.current_version
.as_ref()
.ok_or_else(|| anyhow!("failed to get current version for rustup release"))?;

let manifest_path = self.dl_dir().join("release-stable.toml");
let manifest = format!(
r#"
schema-version = '1'
version = '{}'
"#,
version
);

fs::write(&manifest_path, manifest)?;

run(self
.aws_s3()
.arg("cp")
.arg("--only-show-errors")
.arg(manifest_path)
.arg(&self.s3_artifacts_url("release-stable.toml")))
}
}