Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/release: in -upload mode, difficult to recover/resume if third step fails #45893

Closed
1 task done
dmitshur opened this issue Apr 30, 2021 · 2 comments
Closed
1 task done

x/build/cmd/release: in -upload mode, difficult to recover/resume if third step fails #45893

dmitshur opened this issue Apr 30, 2021 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Apr 30, 2021 •
edited
Loading

When release runs in upload mode (that is, with -upload flag), it goes through 3 high level stages:

  1. uploads files to GCS
  2. waits for edge cache
  3. uploads payloads to golang.org

Before starting, it helpfully checks for gcloud Application Default Credentials and refuses to continue if not (see code here). This is needed for step 1, uploading files to GCS.

A different set of permissions is needed for step 3, uploading payloads to golang.org. Specifically, a valid gomote token that's been added to validUser is needed.

What can currently happen is if release -upload is run and step 3 fails (e.g., due to a missing gomote token), then it is difficult to recover from that state. Rerunning release -upload after fixing the gomote token will fail at step 1, because the files will have been uploaded, and permissions for release managers by default allow them to add files, not overwrite existing ones.

This pitfall can be improved in a few ways:

  • add a check that a gomote token is present before starting (done in CL 315929)
    • this is trivial to implement, and will likely catch most instances
  • consider making it so that release -upload can be restarted safely without needing manual intervention
    • for example, skip over files that have been uploaded successfully instead of attempting to reupload, or modify permissions if it's safe to always allow file overwriting

I'll send a CL to resolve the first bullet point before the next release.

CC @golang/release, @dr2chase.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Apr 30, 2021
@dmitshur dmitshur added this to the Unreleased milestone Apr 30, 2021
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/315929 mentions this issue: cmd/release: check that a gomote token exists before uploading

gopherbot pushed a commit to golang/build that referenced this issue May 3, 2021
@dmitshur
cmd/release: check that a gomote token exists before uploading
ce4d62a 
It's helpful to check that required credentials exist before starting
the upload operation. That way, if the gomote token is missing, the
user will be told right away, rather than the operation failing midway.

For golang/go#45893.

Change-Id: I70aec0a9021a714032f2d0a9a8e51465cefd262e
Reviewed-on: https://go-review.googlesource.com/c/build/+/315929
Trust: Dmitri Shuralyov <[email protected]>
Trust: Alexander Rakoczy <[email protected]>
Reviewed-by: Carlos Amedee <[email protected]>
Reviewed-by: Alexander Rakoczy <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/470755 mentions this issue: cmd/release: deprecate

@dmitshur dmitshur mentioned this issue Jun 21, 2023
Open
@golang golang locked and limited conversation to collaborators Feb 23, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@dmitshur @gopherbot