Stop using the deceiving continue-on-error

[pirj]

Build jobs appear as green even though they actually fail, e.g. #2537 (comment)

A better approach is to mark jobs as "required" in branch protection.

This way, failed jobs are red, but this doesn't prevent the PR from being merged upon at maintainers' discretion.

@JonRowe Can you please set up branch protection so it has 6.1 jobs and below as "Required"?

[pirj]

Green for Rails 6.1 and below, red for Rails 7.0 and edge, as expected.

[JonRowe]

The problem with this, and with making status checks required, is it prevents us from having an "allow failures" mode, which as you can see by the red builds, we need for edge rails and currently unsupported rails versions.

My understanding is setting the builds to required, would actually prevent this PR from being merged for example.

[pirj]

setting the builds to required, would actually prevent this PR from being merged for example.

Yes. I suggest marking those jobs that we know should pass, e.g. currently Rails 6.1 on Ruby 2.7.
Others, like Rails 7 on Ruby 3.1 should not be marked as such until we make sure it's stable.

In any case, you can still merge a PR with a red "required" build job, but GH will ask for an additional confirmation:

A benefit, as I see it, is that non-required builds are shown as red.
This may save us some trouble with regressions in Rails 7 support when we get to it. With the current approach, the build will appear as "all green", even if the PR broke Rails 7 support that was previously green.
An easy way to check this it to unfold jobs with "Show all checks", and check build times for suspiciously low "Successful in 2m" where the baseline is ~15m.

[JonRowe]

A benefit, as I see it, is that non-required builds are shown as red.

Thats not a benefit to me, we want builds to be green if required builds pass, github doesn't give us the option of having optional failures without this, those interested in non-required builds (mostly us as we develop compatibility for things) can investigate individual builds when we care, it would be best if @github allowed us to mark non required builds but thats not a feature they've built yet.