azure: Investigate seemingly spurious failures on CI

[alexcrichton]

Git checkout failures

• arm-android git checkout failure
• armhf-gnu git checkout failure
• dist-various-2 git checkout failure
• even more git checkout failures

Submodule weirdness on Windows

• https://dev.azure.com/rust-lang/rust/_build/results?buildId=295
• https://dev.azure.com/rust-lang/rust/_build/results?buildId=287

Other

• OSX connection lost during a try build
• Build failed to start but marked as successful

[pietroalbini]

• even more git checkout failures

fatal: reference is not a tree: 3b109e51cc8d9a3a833cf7e28db7911a8a0aa503
##[warning]Git checkout failed on shallow repository, this might because of git fetch with depth '2' doesn't include the checkout commit '3b109e51cc8d9a3a833cf7e28db7911a8a0aa503'. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=829603)
##[error]Git checkout failed with exit code: 128

This doesn't seem to be a spurious failure: these jobs started after bors marked the PR as failed due to other failures on Travis/AppVeyor, so the auto branch was already force-pushed to the next PR. We shouldn't see these anymore once we start gating on Azure.

[pietroalbini]

For the submodules weirdness I opened #61322 to investigate.

For the git checkout failures (logs) it seems the error is happening on the Azure side, and there is not much we can do on our end. Any idea on what could have caused it, or what steps we can take to investigate it @rylev @johnterickson?

[alexcrichton]

OSX connection lost during a try build

[johnterickson]

I see two trends - let me know if I'm missing another:

This looks like the auth to GitHub is not working and so git is falling back to a command prompt for creds. Hmm...

##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --tags --prune --progress --no-recurse-submodules --depth=2 origin
fatal: could not read Username for 'https://github.com': terminal prompts disabled
##[warning]Git fetch failed with exit code 128, back off 4.338 seconds before retry.

This sounds vaguely like a race we were seeing where we are getting a notification from GitHub before the commit has been replicated everywhere.

fatal: reference is not a tree: 3b109e51cc8d9a3a833cf7e28db7911a8a0aa503
##[warning]Git checkout failed on shallow repository, this might because of git fetch with depth '2' doesn't include the checkout commit '3b109e51cc8d9a3a833cf7e28db7911a8a0aa503'. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=829603)
##[error]Git checkout failed with exit code: 128

[pietroalbini]

This sounds vaguely like a race we were seeing where we are getting a notification from GitHub before the commit has been replicated everywhere.

@johnterickson I'm pretty sure that's our fault and "expected behavior".

Every time a new build needs to start bors force-pushes the merge commit on the auto branch, thus deleting the previous build's merge commit. If the new build is started shortly after the previous one (for example if someone cancels the build on the bors side to let an higher priority one start) the checkout step of the previous one might have not started yet and it will try to get the old HEAD, causing that error.

I've also seen the error only on builds that ran for ~5 minutes before being killed by our cancelbot because a newer build started, confirming the hypothesis.

[pietroalbini]

There is another spurious failure which is pretty bad: this build failed to "prepare" but was marked as successful on the GitHub Checks.

[johnterickson]

@pietroalbini I raised that "succeeded but actually failed" build with the right team - definitely concerning and I appreciate you reporting it.

[alexcrichton]

Another two that have come up:

• msvc-tools and gnu-tools on https://dev.azure.com/rust-lang/rust/_build/results?buildId=398 show:

+ /tmp/checktools.sh ../x.py /tmp/toolstates.json linux
fatal: ambiguous argument 'HEAD^': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

@pietroalbini or @kennytm do y'all know what might be causing that?

• All of a sudden our script to enable ipv6 in docker is failing. I don't know why this changed all of a sudden but it shouldn't be the hardest thing to work around (testing something on try now)

[kennytm]

Maybe the HEAD commit did not have a parent commit? Is the commit 322afaf associated to any branch during git fetch --depth=2?

[mati865]

For IPv6 error /etc/docker seems to not exist.

[alexcrichton]

Ah ok @kennytm that's probably it. It sounds like a specific git history is expected and/or fetch depth, but I'm just pushing up raw commits which probably breaks the script's assumption. For the failing builds I'm just making manual commits and pushing them to try. Seems like an expected failure.

[johnterickson]

For what it's worth, I had "fixed" this in my original branch in that it would swallow this error.

[pietroalbini]

@ethomson said on Discord the checkout issues are now fixed 🎉

On a related note, GitHub has deployed some fixes that they think will mitigate your sporadic build failures. Could you please @ ping me if you see another beginning today?
Yes, GitHub thinks that they've resolved the issue, but if not, I think that we can add some logic on our side as well.

[ethomson]

Thanks for linking me to this issue, @pietroalbini. Looking at the topic, I'm looking at this section:

Submodule weirdness on Windows
https://dev.azure.com/rust-lang/rust/_build/results?buildId=295
https://dev.azure.com/rust-lang/rust/_build/results?buildId=287

Naive question: what is the weirdness that you're describing? It's not obvious from the logs, because I don't really know what I'm looking for.

[alexcrichton]

Ah yeah sorry I should have been more descriptive there! On build 287 the x86_64-msvc-2 build fails the build due to an odd error message (presumably a submodule missing) and the logs for the "Check out submodules (windows)" step, while successful, are sort of funny. I'm not sure if this is really an Azure issue though, it may have been a case that we got a bad tarball from GitHub and swallowed the error by accident (this is what #61322 was hoping to help diagnose).

The 295 build is similar except for just a different windows builder (dist-x86_64-msvc)

I haven't seen these happen again myself, though, so it may have been just a transient issue. AFAIK most issues we've listed here have been addressed one way or another, so it may actually be time to close this!

[pietroalbini]

By the way, since I couldn't reproduce that failure locally I also added some extra debug info to see what's actually happening there. Didn't get a chance to see the output when the spurious failure happens yet.

[pietroalbini]

@ethomson fyi, those "could not read Username" checkout failures happened again.

[ethomson]

@pietroalbini :( Thanks. Let me see where the fix for that is in the deployment queue and if we expect that your account has it ...

[ethomson]

@pietroalbini Update on 403's ("could not read Username") - that fix was still making its way through progressive deployment. We explicitly enabled it on your account. Please let me know if you see it again, as that means our fix wasn't actually a fix for the issue.

[steveklabnik]

Triage: given that we're not longer using Azure pipelines, I imagine this issue can be closed. Anyone from @rust-lang/infra who can confirm?

[Mark-Simulacrum]

Indeed! Closing.

We're still using macOS on Azure, but I don't think we've seen these failures there recently at least.