Bazel caching for GitHub Actions workflows

[wchargin]

It looks like your CI jobs build from scratch on every build, and thus
take many hours of workflow time (e.g., 7 hours for this workflow).
These cause builds for the entire TF organization to queue.

Bazel has built-in support for incremental remote caching. This is
not a Travis-style giant opaque cache that needs to be sent to and from
the VM on each run. Instead, it’s a fine-grained key-value store that
stores the results for individual actions on a remote server. It has
transparent integration with GCS and can be used with just a backing
bucket.

Could you please investigate using this? It would make you happy, and it
would make us and everyone else happy.

We use this on TensorBoard. It speeds up our builds from ~75 minutes
(clean slate) to ~4 minutes (no-op). You can look at our workflow file
and our bazelrc for guidance.

[yongtang]

@wchargin We will be happy to use bazel cache as this will help speed up our CI process as well. At the moment the biggest blocker is that we don't have a GCS account/credential.

@angerson @ewilderj @martinwicke Wondering if you can help with GCS account/credential to speed up the build?

[angerson]

FYI @theadactyl as well. I'm looking into the technical setup for a TF sub-project for this kind of thing.

[angerson]

@yongtang I started our internal "official" process for setting up a new GCP project to contain a bucket for SIGs to use, but unfortunately there's a production freeze and my request won't be processed until the 5th of January. This should cover due-diligence for any privacy/security related policy problems, although it's still possible we'll run into other nontechnical issues.

FYI @perfinion

[yongtang]

@angerson Thanks for the update 👍

[angerson]

I'm still working on this. It's going slow because there's a complex internal approval process for things like the ability to export service account credentials for bazel.

[angerson]

Another update: I've been working on this with the following assumptions:

1. bazel build can only cache via a GCP service account credential file
2. There is no method that doesn't require a service account credential file

Our company security team can grant an exception allowing us to export service account keys, but they must have a prohibitively short lifespan. I'm now looking at whether or not I can automate a process that generates new keys regularly and automatically pushes them to a GitHub Actions secret.

[kvignesh1420]

Thanks @angerson !

[yongtang]

Added a PR #1287 as a first step to introduce the bazel cache. The PR only adds support for git push commit. The forked PR by outside contributors still do not have access yet. We will see if we can address it later.

[yongtang]

The latest master commit reduces macOS build time from 1 hours 24 mins to 12 mins:

https://github.com/tensorflow/io/runs/1788887957?check_suite_focus=true

[ewilderj]

That's pretty cool!

[wchargin]

That’s awesome—thanks very much to @angerson for setting this all up!

[angerson]

Great!

We're still working on allowing PRs to use the cache, which depends on another policy exemption I'm requesting.

[kvignesh1420]

That would be great! thanks a lot, @angerson.