cmd/pj-rehearse: Truncate extra rehearsals

[wking]

Before this commit, pj-rehearse would just give up:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_release/12959/pull-ci-openshift-release-master-pj-rehearse/1318651994902106112/build-log.txt | tail -n2
time="2020-10-20T20:36:51Z" level=info msg="Created a rehearsal job to be submitted" org=openshift rehearsal-job=rehearse-12959-pull-ci-openshift-kubernetes-master-e2e-cmd repo=release target-job=pull-ci-openshift-kubernetes-master-e2e-cmd target-repo=openshift/kubernetes
time="2020-10-20T20:36:51Z" level=info msg="Would rehearse too many jobs, will not proceed" org=openshift rehearsal-jobs=68 rehearsal-threshold=45 repo=release

But skipping rehearsals for a change that touches tons of jobs, like we have done since ef4f47e (openshift/ci-operator-prowgen#78), makes it possible to break a whole lot of things without failing a warning rehearsal. With this commit, we pivot from "give up and test nothing" to "test as many of the touched jobs as we can afford".

It would be nice to intelligently truncate, e.g. if we touch a few types of jobs, or a few different workflows. But grouping seems complicated, so for this commit I'm just randomly shuffling and then dumping the tail.

CC @petr-muller

[wking]

I've pushed 50505f9 -> f3352ae to fix:

cmd/pj-rehearse/main.go:342:49: undefined: limit 

[wking]

e2e:

error: some steps failed:
  * could not run steps: step [release:custom] failed: the following tags from the release could not be imported to stable-custom after five minutes:
- aws-machine-controllers from registry.svc.ci.openshift.org/origin/4.3-2020-10-19-215406@sha256:367bad806b883bb2531a3c05e9a3782a76d0be3abccf5b47b9d5724da59c3bc9
...

But that seems orthogonal. I think this is ready to land, just needs a new lgtm since my typo fix.

[alvaroaleman]

/lgtm
/retest

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alvaroaleman, stevekuznetsov, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [alvaroaleman,stevekuznetsov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[petr-muller]

I'm not a big fan of this, as this will result in gargantuan bot comments with old stale rehearsal failures - this is one of the most visible annoyances and source of confusion of users, and this PR will make that much worse.

[alvaroaleman]

@petr-muller what about instead sorting them and using the first N?

[wking]

Or pruning stale failures? Since that is an issue, even without this feeding in its churn.

[alvaroaleman]

Or pruning stale failures? Since that is an issue, even without this feeding in its churn.

The standard pruning logic is in prow and not easely changeable because that affects a lot of ppl and I am not fond of the idea of adding a github client and a custom pruning logic to the rehearsal tool

[petr-muller]

Sorting helps but not entirely because the underlying set changes between runs, too

Pruning stale failures... I understand that is a wanted feature of Prow itself, to capture that some job ran (possibly with manual trigger only) in the past revision. With rehearsals it's different and annoying, but it would be quite a lot of work, b/c rehearsal tool does not currently interact with GH API in any way.

[petr-muller]

Bloody hell Alvaro beat me :)

[wking]

Pruning stale failures... I understand that is a wanted feature of Prow itself, to capture that some job ran (possibly with manual trigger only) in the past revision.

Looks like the The following tests failed, say /retest to rerun all failed tests: summary comments include a Commit column today. Maybe we can use that to exclude, or at least more obviously call out, jobs which had run on older commits?

[alvaroaleman]

Maybe we can use that to exclude, or at least more obviously call out, jobs which had run on older commits?

I personally would like that as well (ref kubernetes/community#3621 (comment)) but:

• That discussion is moving really really slow and we need buy in from others
• It means we will need up to one more token per reporting (because we need to find out the current HEAD sha)