Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/relui: build releases #51797

Closed
heschi opened this issue Mar 18, 2022 · 56 comments
Closed

x/build/cmd/relui: build releases #51797

heschi opened this issue Mar 18, 2022 · 56 comments
Assignees
@heschi
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@heschi
Copy link
Contributor

heschi commented Mar 18, 2022

relui should subsume the functionality of x/build/cmd/releasebot and x/build/cmd/release.
@heschi heschi added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Mar 18, 2022
@heschi heschi added this to the Unreleased milestone Mar 18, 2022
@heschi heschi self-assigned this Mar 18, 2022
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/394360 mentions this issue: internal/gcsfs: create

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/394362 mentions this issue: cmd/release: remove -tarball support, no longer needed

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/394361 mentions this issue: cmd/releasebot: remove security release support

gopherbot pushed a commit to golang/build that referenced this issue Mar 22, 2022
@heschi @gopherbot
cmd/releasebot: remove security release support
9adcb24 
As I understand it this path has bitrotted and is no longer supported.
Remove it.

For golang/go#51797.
Closes golang/go#37120.

Change-Id: I6b775f7899abb0fdc7726548b81dcfdba90c69b2
Reviewed-on: https://go-review.googlesource.com/c/build/+/394361
Trust: Heschi Kreinick <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Mar 22, 2022
@heschi @gopherbot
cmd/release: remove -tarball support, no longer needed
746a6d3 
It was only used by security releases.

For golang/go#51797.

Change-Id: I241cfe23797f21daded434b1baa91386097bf8a0
Reviewed-on: https://go-review.googlesource.com/c/build/+/394362
Trust: Heschi Kreinick <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/391074 mentions this issue: cmd/release: drop relocation type checks

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/394715 mentions this issue: cmd/release: remove edge_kick_command

gopherbot pushed a commit to golang/build that referenced this issue Mar 22, 2022
@heschi @gopherbot
cmd/release: drop relocation type checks
c8b6cb0 
1.15 is out of support, and I don't understand the value of the test for
1.16+.

AIUI, we had a problem in 1.15 where we used a newer relocation type
than some toolchains supported, and we wanted to make sure we didn't
accidentally use that type before we were ready. That makes sense to me.
But I don't see the value of guaranteeing that we don't use the old
kind, which everything should still support.

As always, please correct me if I'm missing something -- I don't mind
keeping this if it's actually useful.

Updates golang/go#31293.
For golang/go#51797.

Change-Id: If92e4b3e22759183d856e40b52a45d1251aa8e45
Reviewed-on: https://go-review.googlesource.com/c/build/+/391074
Trust: Heschi Kreinick <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Mar 22, 2022
@heschi
cmd/release: remove edge_kick_command
6ea17dd 
relui won't be able to run random commands. Once this CL is submitted,
I'll update the sync job to run every 10-30 seconds so that we don't
have to wait very long even without the poke, and remove it from our
internal documentation.

For golang/go#51797.

Change-Id: I58f3aeb509f42e7502b398c767d4873618e24e18
Reviewed-on: https://go-review.googlesource.com/c/build/+/394715
Trust: Heschi Kreinick <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/387714 mentions this issue: internal/releasetargets: create

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/394734 mentions this issue: cmd/release,cmd/releasebot: continue migration to releasetargets

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/395536 mentions this issue: cmd/release: move extra CFLAGS into releasetargets

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/395537 mentions this issue: cmd/release: remove Build

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/395934 mentions this issue: cmd/release/releaselet: delete dead code

@heschi
Copy link
Contributor Author

heschi commented Mar 28, 2022

@dmitshur asked me to write down some thoughts on where I'm going with this. I'm still learning the gory details of how all this works, but here's what I'd like to end up with. We want a workflow where we have a reasonable level of parallelism, and where units of work are on the smaller side so we lose the smallest amount of work on a failure. So:

  • Build a source tarball: add VERSION, remove unwanted files.
  • For each release target:
    • Unpack the source tarball. Run make.bash to produce a binary distribution tarball. Remove unwanted files.
      • For cross-compilations, do fixups and stop here.
    • Test the binary distribution: run run.bash so we don't have to rebuild the toolchain from scratch.
      • Currently we run tests in both -short and -long; I don't see a reason to do that, so this should be a single task per target running on the longtest builder if present.
    • For Windows, while the tests are running:
      • Convert the tarball to a .zip for the binary archive. (I think it's nice to keep this out of the make step, which is otherwise platform-neutral.)
      • Build an MSI.

Note that each of these steps consumes 0-1 files and produces 0-1 files, so we have a very clear flow.

cc @golang/release

@dmitshur dmitshur mentioned this issue Mar 29, 2022
Closed
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/396437 mentions this issue: cmd/release: only use flags in main

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/395935 mentions this issue: cmd/release: refactor release build process

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/396554 mentions this issue: cmd/release: continue refactoring build code

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/397477 mentions this issue: cmd/release: refactor to separate steps, combine long test builds

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/398498 mentions this issue: all: extract build steps to internal/task

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/398694 mentions this issue: cmd/relui: add build release workflow

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/399158 mentions this issue: internal/task: fix bugs in release tasks

@dmitshur dmitshur mentioned this issue Apr 13, 2022
Closed
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/401338 mentions this issue: [dev.boringcrypto] misc/boring: skip long tests during build.release

gopherbot pushed a commit to golang/build that referenced this issue Jun 9, 2022
@toothrot
intenal/relui/migrations: drop problematic workflow
c39906c 
Removes unrecoverable workflow.

For golang/go#51797

Change-Id: I0dbd7e61e3d1fb3ac3708a76aad39c753ddb5531
Reviewed-on: https://go-review.googlesource.com/c/build/+/411060
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Alex Rakoczy <[email protected]>
Reviewed-by: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/411063 mentions this issue: cmd/relui: fix website upload username

gopherbot pushed a commit to golang/build that referenced this issue Jun 10, 2022
@heschi @gopherbot
cmd/relui: fix website upload username
09064db 
Unlike the coordinator, the website doesn't want the user- prefix on the
username.

For golang/go#51797.

Change-Id: I169bdf1dd8ffd7cad6fccb2cf752be4271efe094
Reviewed-on: https://go-review.googlesource.com/c/build/+/411063
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/411064 mentions this issue: cmd/relui: actually fix user name

gopherbot pushed a commit to golang/build that referenced this issue Jun 10, 2022
@heschi @gopherbot
cmd/relui: actually fix publishing
bf85695 
I got the arguments to TrimPrefix backward, but more importantly,
u.Query() returns a temporary copy.

For golang/go#51797.

Change-Id: I37a9df2adf47f1d72236d02761989d62e9991fc7
Reviewed-on: https://go-review.googlesource.com/c/build/+/411064
Reviewed-by: Cherry Mui <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/411894 mentions this issue: internal/workflow: always use JSON-marshaled task results

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/411897 mentions this issue: internal/task: skip CLs that don't make changes

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/411895 mentions this issue: internal/relui: set a deadline on artifact presence checks

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/411896 mentions this issue: gerrit: switch to a generic NotExist error

gopherbot pushed a commit to golang/build that referenced this issue Jun 13, 2022
@heschi
internal/relui: set a deadline on artifact presence checks
c3d2f80 
We saw some signs of requests hanging when we tried it out. Make sure we
don't get stuck forever.

For golang/go#51797.

Change-Id: I24994105beee643b915dd17d520eb08843b0d864
Reviewed-on: https://go-review.googlesource.com/c/build/+/411895
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/411902 mentions this issue: internal/relui: use a per-workflow directory for signing

gopherbot pushed a commit to golang/build that referenced this issue Jun 13, 2022
@heschi @gopherbot
internal/workflow: always use JSON-marshaled task results
512b5f0 
We only realized JSON marshaling of the release tasks was broken when
we retried them because in normal operation we used the in-memory
versions. Change to always use the marshaled results.

For golang/go#51797.

Change-Id: I9dbca62520732a7b00d64dc2481ab35b221fecf0
Reviewed-on: https://go-review.googlesource.com/c/build/+/411894
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Jun 14, 2022
@heschi
gerrit: switch to a generic NotExist error
c2ca09d 
...and notice that only the one I just added is being used...

For golang/go#51797.

Change-Id: If62d643c2a0524231e88f1cde2cd4c980e63dd57
Reviewed-on: https://go-review.googlesource.com/c/build/+/411896
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Jun 14, 2022
@heschi @gopherbot
internal/task: skip CLs that don't make changes
5227dc8 
In the most recent release we retried the DL CL task, and it failed
because Gerrit rejected the no-op changes. Change the Gerrit client to
return no change in this case, and the surrounding tasks to tolerate it.

If this happens for the VERSION CL we will tag whatever the branch head
happens to be at the time, which seems correct to me.

For golang/go#51797.

Change-Id: Ieb69a4c96c7ebe057a53d0fca4d713dca0fc6f47
Reviewed-on: https://go-review.googlesource.com/c/build/+/411897
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
Reviewed-by: Alex Rakoczy <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/412176 mentions this issue: internal/relui: simplify rendering of homepage

gopherbot pushed a commit to golang/build that referenced this issue Jun 15, 2022
@toothrot
internal/relui: simplify rendering of homepage
2838fbb 
This refactors the homepage rendering to use a nested template for each
task row. This will help simplify the template as we add more complex
layout to the outputs of workflows and tasks.

Updates golang/go#51797
Updates golang/go#40279
For golang/go#53382

Change-Id: I85a86b82bdc79c7fb4e837d884af922c7028295d
Reviewed-on: https://go-review.googlesource.com/c/build/+/412176
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Alex Rakoczy <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Jun 17, 2022
@heschi
internal/relui: use a per-workflow directory for signing
1503a12 
We learned the hard way that using a well-known directory creates
confusion in the release process. Instead, create a "signing" directory
inside the per-workflow scratch dir and use that to communicate with the
internal signing tool.

In addition, create a sentinel file "ready" that can be used to signal
the internal tool to start working.

For golang/go#51797.

Change-Id: Id8236b932e22c445cc03ecec3e975d85583c914b
Reviewed-on: https://go-review.googlesource.com/c/build/+/411902
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/412390 mentions this issue: internal/relui: write to "ready" file

gopherbot pushed a commit to golang/build that referenced this issue Jun 17, 2022
@heschi @gopherbot
internal/relui: write to "ready" file, fix signing arg
d6ffeea 
Simply opening and closing a file isn't enough to create it.

Also, don't use path.Join on URLs: it converts gs://foo to gs:/foo which
is a very different URL.

For golang/go#51797.

Change-Id: Ic681e151f2a2258f93e055564589d4440358e954
Reviewed-on: https://go-review.googlesource.com/c/build/+/412390
Auto-Submit: Heschi Kreinick <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/416219 mentions this issue: internal/task: don't check blockers for RC releases

@heschi
Copy link
Contributor Author

heschi commented Jul 6, 2022

I'm gonna call this done. We no longer use cmd/releasebot or cmd/release in the standard Go release process and we've had reasonable success using it.

@heschi heschi closed this as completed Jul 6, 2022
gopherbot pushed a commit to golang/build that referenced this issue Jul 7, 2022
@heschi @gopherbot
internal/task: don't check blockers for RC releases
3bb68c2 
I missed a quirk of logic in releasebot.

For golang/go#51797.

Change-Id: I8f1fd9f4af4ae6743d7d874b5223b45fd9127c96
Reviewed-on: https://go-review.googlesource.com/c/build/+/416219
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/418789 mentions this issue: internal/relui: choose major versions dynamically

gopherbot pushed a commit to golang/build that referenced this issue Jul 25, 2022
@heschi
internal/relui: choose major versions dynamically
4d3a945 
For convenience, I originally hardcoded Go1.17, 18, and 19 as the live
major Go versions. We're about to release 1.19, so that's not so
convenient any more.

Extract a function to read the live major version, and use it to
choose what workflows to create. While I'm here, add workflows to do
a single minor release in case we need to do that at some point.

For golang/go#51797.

Change-Id: I1e73e42b6c78df7f3e8deb0a191893ff3c887297
Reviewed-on: https://go-review.googlesource.com/c/build/+/418789
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
@heschi heschi moved this to Done in Go Release Sep 27, 2022
@golang golang locked and limited conversation to collaborators Jul 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@heschi heschi

Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@gopherbot @heschi