[CI] Use GCS buckets for bazel remote caching

[brianseeders]

TLDR: Use GCS buckets for bazel remote caching in CI for a cheap and easy bootstrap speed boost. Local remote cache is currently unchanged. All steps can read and write.

Bootstrap times in CI vary a lot over the course of the day, and can get pretty long on smaller machines when changes are made that invalidate the local cache inside the agent images (which are refreshed daily in the morning). We would like to enable caching across CI in a performant and cost-effective way.

I trialed:

• Using bazel-remote, with grpc, hosted on an instance in our GCP project
• Using a single, "multi-regional" GCS bucket located in the U.S.
• Using single-region GCS buckets in every GCP region where we run CI (there are currently 5).

bazel-remote notes:

• Probably the most expensive and least performant option, at least as I had it configured. Worse performance for instances further from where the service was hosted
• Could possibly host a separate instance in each region to gain performance across regions, but it will be 5x as expensive
• Does not have an HA solution
• Would have to be hosted, maintained, upgraded, and monitored by us
• With 100 instances running bootstrap starting from 0 cache (a worst-case scenario), CPU load spiked to around 4
• Example with 100 workers, no disk cache, full remote cache - 2min-4min for bootstrap, depending on the region of the agent. Has a lot of variability inside the same region as well

GCS single bucket:

• Uses https instead of grpc
• Was faster than bazel-remote for worst-case scenario
• Similar to bazel-remote, it's much slower for instances not close to the U.S.
• Cheap and zero maintenance
• File retention set to 48 hours, as we only need to cache objects not present in local cache, which is updated every 24 hours
• Example with 100 workers, no disk cache, full remote cache - 2min-4min for bootstrap, depending on the region of the agent

GCS bucket-per-region:

• Same as GCS single bucket, except:
• Storage cost will be 5x as much as single bucket, but it's a pretty small cost. I'm not sure how much storage 48 hours worth of objects will be (it should be pretty small, probably in MB), but 1TB is about $20/mo.
• Objects have to be cached separately across all the regions, but will generally happen during the on-merge job by jest jobs, and FTR jobs if still missing by then
• All regions are fast
• Bandwidth costs should be smaller as objects are stored in the same region
• Example with 100 workers, no disk cache, full remote cache - about 2min for bootstrap

Given all of this, the last option (GCS bucket-per-region) seems like the best choice. This may change in the future when we're utilizing bazel for even more, and we can always reassess.

As a side note: We could also use this for local remote cache if we wanted. We just wouldn't get the UIs, statistics, historical tracking, etc. that Buildbuddy provides.

[brianseeders]

buildkite build this

[brianseeders]

buildkite build this

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[spalger]

🎉🎉🎉 This is awesome!! 🎉🎉🎉

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: a041894
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

✅ unchanged

History

• 💚 Build #42212 succeeded 3d8d112
• 💚 Build #42194 succeeded a63ba24
• 💚 Build #42190 succeeded a63ba24
• 💚 Build #42153 succeeded d3e1a88
• 💚 Build #41906 succeeded 8ff132b
• 💚 Build #41903 succeeded abfb676

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[mistic]

I believe it is also worth it to try https://github.com/znly/bazel-cache in the future

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.2
✅	7.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation