Retry invocation if Bazel exits with code 39

[coeuvre]

With bazelbuild/bazel#17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With bazelbuild/bazel#17462, Bazel is able to continue the build without bazel clean or bazel shutdown.

This PR makes bazelisk automatically retries the invocation if it sees exit code 39 from Bazel, i.e. "rewinding" on a larger scale.

[coeuvre]

cc @tjgq @meisterT

[brentleyjones]

This will be a very nice change to go along with the Bazel 6.1 changes.

[meteorcloudy]

I wonder if we should guard this behaviour behind a flag? It could be a bit surprising for users that Bazelisk is automatically retrying Bazel invocation.

Also can we add some documentation and ideally a test for this use case?

[meteorcloudy]

We should probably print out some message notifying Bazelisk is retrying the bazel invocation?

[coeuvre]

The exit code 39 is new in Bazel and is guarded by flag in 6.1.0. I would rather not add a new flag here because we certainly want to retry on 39. We can update documentation to mention this behaviour though.

Adding test is tricky. It's hard to make Bazel exits 39 without directly controlling the remote CAS. We have integration test in Bazel with the help of remote worker. I don't think there is an easy way to mimic it here.

Using a mock also requires code refactor around runBazel (e.g. passing a runner interface). If it sounds good, I can go ahead and add a unit test with mock.

[meteorcloudy]

Thanks for the context.

If it sounds good, I can go ahead and add a unit test with mock.

Yes, that sounds good enough!

[fweikert]

I wouldn't complain about a test either :)

[coeuvre]

Per internal discussion, we probably want to include the retry in Bazel itself. Closing for now.