don't error on missing results*.json

[IanButterworth]

Ref: https://github.com/buildkite-plugins/test-collector-buildkite-plugin#missing-error-integer

cc. @KristofferC

[IanButterworth]

Can we make Test not Required?

Generally I think this repo might just better suit requiring 1 other approval, rather than required runs, given that the julia repo could be broken. (I'm expecting the windows run to timeout again, in 3h, which is tedious).

Or can someone admin merge.

[IanButterworth]

@DilumAluthge perhaps?

[DilumAluthge]

I've admin merged.

[DilumAluthge]

I'm reluctant to relax the required status checks here, just because there have been cases where a PR to this repo looks good, and it is merged with incomplete CI results, but then breaks JuliaLang/julia. In this case, JuliaLang/julia was good, and then the bad PR merged in this repo breaks JuliaLang/julia.

But then as @IanButterworth points out, the counter-case is when JuliaLang/julia is already broken, and thus a fast merge is needed here to fix JuliaLang/julia.

[DilumAluthge]

The problem with removing Test from the list of required status checks is that it will have no impact unless the summary status (buildkite/julia-buildkite) is also removed from the list of required status checks.

And I'm not thrilled about the idea of removing buildkite/julia-buildkite from the list of required status checks. We had a very interesting case a while back where all Buildkite statuses were green on a PR except for the buildkite/julia-buildkite status. We couldn't figure out why, and we were all tempted to just merge the PR, because we assumed the buildkite/julia-buildkite status was being flaky or something. Eventually I managed to diagnose it - there was an actual bug in the PR (that would have broken CI on JuliaLang/julia), and the only way we detected it was by having buildkite/julia-buildkite required.

It's definitely a tough balance: on one hand, I want to avoid introducing new breakage to JuliaLang/julia, but on the other we need to be able to quickly fix existing breakage on JuliaLang/julia.

Perhaps we can keep the required status checks as-is, and maybe expand the pool of people that can admin-merge when there is a time-sensitive PR (like this one) that needs to be merged with incomplete CI checks?