x/build/{cmd/coordinator,app/appengine}: combine codebases #34744
Comments
dmitshur
added
Builders
|
I regret that I have but four emojis to give to this bug. |
|
Change https://golang.org/cl/199800 mentions this issue: |
The filtering of builders in commitBuilders was too strict, causing post-submit builders that are configured to run on release-branches only to not get included in the list of builders. This change makes the following builders no longer skipped: darwin-amd64-10_10 freebsd-386-10_3 freebsd-386-10_4 freebsd-386-11_1 freebsd-amd64-10_3 freebsd-amd64-10_4 freebsd-amd64-11_1 nacl-386 nacl-amd64p32 We may need to adjust the build.golang.org UI after to avoid including unhelpful columns, but this is a first step to get nacl builders to run on release branches again. Updates golang/go#34738 Updates golang/go#34744 Change-Id: Iaf2b93aedd5f44b48b9a63b57f12549fe50b1637 Reviewed-on: https://go-review.googlesource.com/c/build/+/199800 Reviewed-by: Brad Fitzpatrick <[email protected]>
|
Change https://golang.org/cl/199878 mentions this issue: |
…branch This is a follow-up to CL 199800. That change started to consider more builders as active to allow release-branch-only builders to run. However, it also made those builders show up as new empty columns when viewing builds for the master branch. There are quite a few old FreeBSD builders that only run on release-branch.go1.12, and it's too disruptive to have them appear everywhere. So, hide them when viewing the master branch of Go repo in the UI. There can be more UI improvements to be made, and this can become too much of a whack-a-mole to address them one by one. The scope of this CL is to fix the most disruptive high priority problem for now. Further improvements will happen later, with merging of app/appengine and cmd/coordinator codebases in mind. Updates golang/go#34738 Updates golang/go#34744 Change-Id: I3df75f8b2bbd5f6fe8097c181ee8a1b1b4354dc9 Reviewed-on: https://go-review.googlesource.com/c/build/+/199878 Run-TryBot: Dmitri Shuralyov <[email protected]> Reviewed-by: Bryan C. Mills <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>
The filtering of builders in commitBuilders was too strict, causing post-submit builders that are configured to run on release-branches only to not get included in the list of builders. This change makes the following builders no longer skipped: darwin-amd64-10_10 freebsd-386-10_3 freebsd-386-10_4 freebsd-386-11_1 freebsd-amd64-10_3 freebsd-amd64-10_4 freebsd-amd64-11_1 nacl-386 nacl-amd64p32 We may need to adjust the build.golang.org UI after to avoid including unhelpful columns, but this is a first step to get nacl builders to run on release branches again. Updates golang/go#34738 Updates golang/go#34744 Change-Id: Iaf2b93aedd5f44b48b9a63b57f12549fe50b1637 Reviewed-on: https://go-review.googlesource.com/c/build/+/199800 Reviewed-by: Brad Fitzpatrick <[email protected]>
…branch This is a follow-up to CL 199800. That change started to consider more builders as active to allow release-branch-only builders to run. However, it also made those builders show up as new empty columns when viewing builds for the master branch. There are quite a few old FreeBSD builders that only run on release-branch.go1.12, and it's too disruptive to have them appear everywhere. So, hide them when viewing the master branch of Go repo in the UI. There can be more UI improvements to be made, and this can become too much of a whack-a-mole to address them one by one. The scope of this CL is to fix the most disruptive high priority problem for now. Further improvements will happen later, with merging of app/appengine and cmd/coordinator codebases in mind. Updates golang/go#34738 Updates golang/go#34744 Change-Id: I3df75f8b2bbd5f6fe8097c181ee8a1b1b4354dc9 Reviewed-on: https://go-review.googlesource.com/c/build/+/199878 Run-TryBot: Dmitri Shuralyov <[email protected]> Reviewed-by: Bryan C. Mills <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>
|
Change https://golang.org/cl/208319 mentions this issue: |
|
Change https://golang.org/cl/208320 mentions this issue: |
This was the performance/benchmark code from ~three generations ago. It's unused and unmaintained. It broke when we moved from mercurial to git, IIRC. I'm attempting to modernize this code (for golang/go#34744) but it'd be easier if there's less code to deal with. Updates golang/go#34744 Change-Id: Ib4999830b05df9ffad9b46964022325404350b47 Reviewed-on: https://go-review.googlesource.com/c/build/+/208319 Reviewed-by: Bryan C. Mills <[email protected]> Reviewed-by: Andrew Bonventre <[email protected]>
… tool This moves the /key handler to generate build keys to a standalone tool. The old way has been largely broken for years (when using the build.golang.org domain) due to internal App Engine changes. We have to access it via https://build-dot-golang-org.appspot.com/key instead to get authenticated. Also, the App Engine go112 runtime doesn't support authenticated handlers, so more reason to move off App Engine. This CL is part of a series to move off the the build.golang.org App Engine app that mirrors the git history into Datastore Entities, which is full of complication and bugs. These early steps are about removing a bunch of code from the App Engine app so the important bits are easy to see and refactor. Updates golang/go#34744 Change-Id: Iaf8e2bf458b5fea45bf05026d8a6eaf0ead88ec2 Reviewed-on: https://go-review.googlesource.com/c/build/+/208320 Reviewed-by: Bryan C. Mills <[email protected]> Run-TryBot: Bryan C. Mills <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>
|
Change https://golang.org/cl/208322 mentions this issue: |
It hasn't worked in years. This is part of a series of CLs to clean up the build.golang.org App Engine app in prep for it to be modernized, refactored, and replaced. Updates golang/go#34744 Updates golang/go#12509 Change-Id: I9f8445046961ccbe97f7b9c85c0772393bf7d547 Reviewed-on: https://go-review.googlesource.com/c/build/+/208322 Reviewed-by: Bryan C. Mills <[email protected]> Reviewed-by: Andrew Gerrand <[email protected]>
|
Change https://golang.org/cl/208324 mentions this issue: |
This is part of a series of CLs to clean up the build.golang.org App Engine app in prep for it to be modernized, refactored, and replaced, starting with deleting dead code. Updates golang/go#34744 Change-Id: I6cddbb44a63597a308f1d4399d21e5b70a8d83bf Reviewed-on: https://go-review.googlesource.com/c/build/+/208324 Reviewed-by: Andrew Gerrand <[email protected]>
|
Change https://golang.org/cl/208397 mentions this issue: |
This is part of a series of CLs to clean up the build.golang.org App Engine app in prep for it to be modernized, refactored, and replaced, starting with deleting dead code. Some of this was missed in CL 208319. Updates golang/go#34744 Change-Id: I35adf4296b849dabed9d974e7eb58eeedc7d4293 Reviewed-on: https://go-review.googlesource.com/c/build/+/208397 Reviewed-by: Andrew Gerrand <[email protected]>
|
Change https://golang.org/cl/208678 mentions this issue: |
|
Change https://golang.org/cl/208697 mentions this issue: |
… from init tagHandler isn't used by anything. Delete. Less code to update. And document who calls the other handlers. Also, remove the auth requirement from the /init handler. It's harmless to call without auth, and the login restriction went away with the Go 1.12+ runtime anyway, so deleting it gets us closer to being able to use the Go 1.12/Go 1.13 runtimes. (The plan is to delete most the code, port the small remaining bit to the cloud.google.com/go libraries so we can update to Go 1.12/Go 1.13+, and then at that point, since the cloud.google.com/go code will run anywhere, we can just run it in the same process as the coordinator.) Updates golang/go#34744 Change-Id: I929c70945a3e9e27b38b1d5899c7860470361927 Reviewed-on: https://go-review.googlesource.com/c/build/+/208678 Reviewed-by: Bryan C. Mills <[email protected]>
Historically, the build.golang.org was the entire build system, and it maintained a parallel copy of that git history in its datastore. It was always buggy and incomplete and things like force pushes were scary because the datastore mirror could get out of sync. It was also a lot of code to support that sync. This changes build.golang.org to instead get the git history from maintnerd, and then we can remove all the HTTP handlers around updating it, and can remove all the gitmirror code to call it to maintain it. Now build.golang.org only keeps build results, keyed on the commit hash. It's much less code, but is still an App Engine app for now. (but it's getting small enough, that porting it to cloud.google.com/go/datastore is looking very simple) This also adds a new "repos" package to unify the configuration of the various Go repos. There were incomplete & redundant copies all over the place. Updates golang/go#34744 Fixes golang/go#35828 Fixes golang/go#31236 (New branch=mixed support adds this when desired) Fixes golang/go#35944 Change-Id: Ifb39417287df3dea052ba8510566d80b4bc75d51 Reviewed-on: https://go-review.googlesource.com/c/build/+/208697 Reviewed-by: Bryan C. Mills <[email protected]>
|
This is getting pretty close now. Here's a suggested roadmap for finishing this:
|
|
Change https://golang.org/cl/210838 mentions this issue: |
… avoid memcache This implements bullets 4 and 5 from the plan to unify the dashboard & coordinator: golang/go#34744 (comment) Previously the coordinator would POST to the dashboard regularly, for each active build, to say "I'm still working on this build! Write that to memcache!". And then if somebody loaded https://build.golang.org/ it would do a big memcache multi-get to populate the little blue gopher links to their status pages on the coordinator. This instead turns it around. We no longer POST from the coordinator to the dashboard, and we no longer use any memcache (which also means migrating to the App Engine Go 1.13 runtime is easier, which drops the built-in memcache support). Instead, the dashboard now does a GET to the coordinator to get the list of active builds. This also adds test coverage, which we didn't have before. Updates golang/go#34744 Change-Id: I97a486ec362a7a00d29076c81a88d6417b138c1b Reviewed-on: https://go-review.googlesource.com/c/build/+/210838 Reviewed-by: Alexander Rakoczy <[email protected]> Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>
|
Change https://golang.org/cl/210977 mentions this issue: |
|
Change https://golang.org/cl/340589 mentions this issue: |
The build.golang.org service is undergoing a transition as part of https://golang.org/issue/34744#issuecomment-890076711. It's taking some time for the DNS change to propagate and HTTPS certificate to be acquired. During this time, use farmer-ui-test.golang.org as the dashboard URL, since it's currently used by cmd/coordinator to find post-submit work and report post-submit build results. For golang/go#34744. Change-Id: I7ca99dc75afa2960f831248110c9e5053ee95f99 Reviewed-on: https://go-review.googlesource.com/c/build/+/340589 Trust: Dmitri Shuralyov <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>
In htmlView.ServeDashboard, we know very well that the content being served is HTML. Set the Content-Type header explicitly to indicate that, to avoid clients from having to do needless computationally- expensive guessing. (This will help the next change, where we need to know the Content-Type to be able to rewrite URLs in said HTML.) Also improve and inline the logErr helper that was undocumented and used in only 3 places. It's more readable to use http.Error. For golang/go#34744. Change-Id: I783d6559274bd2b9eeb7efcfe9464122cf4a9a68 Reviewed-on: https://go-review.googlesource.com/c/build/+/340431 Trust: Dmitri Shuralyov <[email protected]> Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>
|
Flipping the DNS has been done (in a somewhat bumpy way, see #47580, but it's resolved now). https://build.golang.org is now served by coordinator! Coordinator needed additional permissions to be able to write build results, which has also been resolved by now. The stack up to CL 340434 is mostly reviewed but awaiting a few more reviews. We should be able to proceed with it landing it, making sure everything still works, and then this issue can be considered resolved. |
The coordinator now serves both farmer.golang.org and build.golang.org. It has been possible to access the build dashboard page during local development either by modifying source code, or /etc/hosts, but this is not convenient. Instead, use a hostPathHandler in local development mode which maps: https://localhost:8119/farmer.golang.org/ → farmer.golang.org pages https://localhost:8119/build.golang.org/ → build.golang.org pages Which is very convenient, and will become even better after the next CL unifies the header across the pages. This is based on the approach in golang.org/x/website/cmd/golangorg for serving multiple domains on localhost, implemented in CL 328013 by Russ. For golang/go#34744. Change-Id: Icd90c776db0898a797455cbb0dfb0fc038d050d2 Reviewed-on: https://go-review.googlesource.com/c/build/+/340432 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Trust: Dmitri Shuralyov <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]>
A header is displayed on 3 pages: • farmer.golang.org - the coordinator status page • farmer.golang.org/builders - list of builders • build.golang.org - the build dashboard page (now served by coordinator) They look slightly different visually. There's also no good way to navigate from the dashboard back to coordinator status page. This change makes them all look the same, making it easier to refactor them in the future, and improves ability to navigate. Apply the same change to the work-in-progress dashboard V2 page, to keep it in sync. For golang/go#34744. Fixes golang/go#47477. Change-Id: Ib17d3e2955fa42afbb4bccb249ec5a8786ceab6e Reviewed-on: https://go-review.googlesource.com/c/build/+/340116 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Trust: Dmitri Shuralyov <[email protected]> Trust: Alexander Rakoczy <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>
…ui-test The farmer-ui-test domain was added to help with a gradual rollout of the dashboard unification. It was also used when the build.golang.org DNS was still propagating and not yet fully available. All that's done by now, so these aren't needed and can be removed. Switch to using the final post-unification URL, build.golang.org in production and build-staging.golang.org in the staging environment. Simplify the handler registration by using http.ServeMux's host prefix support. For golang/go#34744. Updates golang/go#47580. Change-Id: I03c0529bd948e812ccf5c5aa312de247d174ad0a Reviewed-on: https://go-review.googlesource.com/c/build/+/340433 Trust: Dmitri Shuralyov <[email protected]> Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]>
The appengine command is no longer needed since the coordinator is serving the build dashboard. The key package is now unused. For golang/go#34744. Change-Id: I74218bac6b7457434975d13842e59e150fc078be Reviewed-on: https://go-review.googlesource.com/c/build/+/336792 Trust: Dmitri Shuralyov <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>
Update documentation to reflect changes from the dashboard unification. For golang/go#34744. Change-Id: Ifb61207dd61496df0e8e9c96fd05730399e3a4b8 Reviewed-on: https://go-review.googlesource.com/c/build/+/340434 Trust: Dmitri Shuralyov <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>
Every now and then, it's highly desirable to be able to view build logs for a build even if it was successful. Those logs are kept as of CL 198197, but they just haven't been linked. At long last, add those links to the build dashboard(s)! 🎉 Keep the visual style of "ok" cells the same, aside that the mouse cursor becomes a hand with an index finger, rather than a text caret. This means this feature isn't very discoverable, but we can address that in the future. (I've experimented with giving the "ok" links an underline, but even a faint underline quickly leads to a large amount of overwhelming visual noise when there are hundreds of "ok" cells on many pages.) Apply the same change to the work-in-progress dashboard V2 page, to keep it in sync. (There are additional CSS changes to make it look more consistent with the current build.golang.org dashboard.) For golang/go#34119. Updates golang/go#34744. Change-Id: I27530e4103bbfd10e8cf09ae1a1cfce06c9bd748 Reviewed-on: https://go-review.googlesource.com/c/build/+/340435 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Trust: Dmitri Shuralyov <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]>
|
This is now live and submitted to x/build. Notably: CL 340433 reconfigured coordinator to use the build.golang.org URL as the dashboard. CL 340435 was a cherry on top, bringing a small enhancement to the now-served-by-coordinator build dashboard. Filed #47630 for some trivial clean up that can happen (after waiting a bit). |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
dmitshur
added
NeedsFix
We've discussed and agreed wanting to merge
app/appengineintocmd/coordinator. The motivation is to have a single codebase, share more code, fewer separate deployments, and tests with more inclusive coverage.I didn't find an existing issue, so making this one to track the task.
/cc @bradfitz @andybons @toothrot
The text was updated successfully, but these errors were encountered: