go_download_sdk: download release table from golang.org/dl

[jayconrod]

If the sdks attribute is not set, go_download_sdk will now download a
list of available releases from https://golang.org/dl. If the version
attribute is not set, go_download_sdk will pick the highest available
version (and Bazel will tell users to set the version attribute for a
reproducible build). Even if version is set, we need to download the
release list in order to get the SHA-256 sums. Bazel won't cache
downloads without an SHA-256 sum.

go_register_toolchains now requires the version argument (previously
named go_version, which is still allowed) if no toolchain was declared
earlier.

With this change, rules_go no longer requires an update after a new Go
version is released.

Fixes #2405

[justaugustus]

This is AMAZING, @jayconrod, and will make my life easier updating Golang in the Kubernetes project!

Will we be able to support pulling in Golang pre-releases as a result of this change?

[jayconrod]

This is AMAZING, @jayconrod, and will make my life easier updating Golang in the Kubernetes project!

@justaugustus Glad to help! Mostly looking for ways to reduce toil.

Will we be able to support pulling in Golang pre-releases as a result of this change?

I think so. https://pkg.go.dev/golang.org/x/website/internal/dl has the doc on the endpoint I'm querying to list release file names and SHA-256 sums. It says unstable releases should be included. There aren't actually any unstable releases at the moment, so I can't test that.

Even if that doesn't work, you can already pull in unstable releases by setting the sdks attribute in go_download_sdk. That's kind of a pain though. With this change, you should be able to set version = "1.16beta1" or something like that in go_register_toolchains and it will do the right thing.

[justaugustus]

Awesome! I actually borrowed some code from you to accomplish this:

https://github.com/kubernetes/repo-infra/blob/a14fc32081dd161ccf68be85536332d6f7882e1e/go/sdk_versions.bzl#L15

It'll be nice to test this out in the next Golang prerelease. Thanks again!

ref: kubernetes/repo-infra#202

[swsnider]

This seems to have broken go sdk extraction again by removing the workaround for bazelbuild/bazel#7055 -- I get the following output using bazel 3.7.1 and rules_go 0.25.0:

INFO: Repository go_sdk instantiated at:
  /Volumes/Projects/repo/WORKSPACE:61:16: in <toplevel>
  /Users/swsnider/.cache/bazel/_bazel_swsnider/external/io_bazel_rules_go/go/private/sdk.bzl:129:21: in go_download_sdk
Repository rule _go_download_sdk defined at:
  /Users/swsnider/.cache/bazel/_bazel_swsnider/external/io_bazel_rules_go/go/private/sdk.bzl:116:35: in <toplevel>
INFO: Repository 'go_sdk' used the following cache hits instead of downloading the corresponding file.
 * Hash '3918e6cc85e7eaaa6f859f1bdbaac772e7a825b0eb423c63d3ae68b21f84b844' for https://doesnotexistipromise.local/go_sdk-linux.tar.gz
If the definition of 'go_sdk' was updated, verify that the hashes were also updated.
ERROR: An error occurred during the fetch of repository 'go_sdk':
   Traceback (most recent call last):
	File "/Users/swsnider/.cache/bazel/_bazel_swsnider/external/io_bazel_rules_go/go/private/sdk.bzl", line 100, column 16, in _go_download_sdk_impl
		_remote_sdk(ctx, [url.format(filename) for url in ctx.attr.urls], ctx.attr.strip_prefix, sha256)
	File "/Users/swsnider/.cache/bazel/_bazel_swsnider/external/io_bazel_rules_go/go/private/sdk.bzl", line 180, column 29, in _remote_sdk
		ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /Users/swsnider/.cache/bazel/_bazel_swsnider/external/go_sdk/temp7406720697368585321/go_sdk-linux.tar.gz to /Users/swsnider/.cache/bazel/_bazel_swsnider/external/go_sdk/temp7406720697368585321: /Users/swsnider/.cache/bazel/_bazel_swsnider/external/go_sdk/test/fixedbugs/issue27836.dir/?foo.go (Input/output error)
ERROR: Analysis of target '[REDACTED]' failed; build aborted: java.io.IOException: Error extracting /Users/swsnider/.cache/bazel/_bazel_swsnider/external/go_sdk/temp7406720697368585321/go_sdk-linux.tar.gz to /Users/swsnider/.cache/bazel/_bazel_swsnider/external/go_sdk/temp7406720697368585321: /Users/swsnider/.cache/bazel/_bazel_swsnider/external/go_sdk/test/fixedbugs/issue27836.dir/?foo.go (Input/output error)
INFO: Elapsed time: 196.138s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
    Fetching /Users/swsnider/.cache/bazel/_bazel_swsnider/external/go_sdk; Extracting /Users/swsnider/.cache/bazel/_bazel_swsnider/external/go_sdk/temp7406720697368585321/go_sdk-linux.tar.gz 169s

My go_download_sdk in WORKSPACE looks like

go_download_sdk(
    name = "go_sdk",
    #    version = "1.15.6",
    sdks = {
        "darwin_amd64": ("go_sdk-darwin.tar.gz", "940a73b45993a3bae5792cf324140dded34af97c548af4864d22fd6d49f3bd9f"),
        "linux_amd64": ("go_sdk-linux.tar.gz", "3918e6cc85e7eaaa6f859f1bdbaac772e7a825b0eb423c63d3ae68b21f84b844"),
    },
    urls = ["https://doesnotexistipromise.local/{}"],
)

[jayconrod]

@swsnider Please open a new issue.

[swsnider]

Opened #2771