Allow users to generate periodic jobs

[stevekuznetsov]

In addition to generating presubmits and postsubmits, a new field on the
test type allows users to generate periodic jobs. In order to support
generating periodic jobs that require the src-build step, we add the
repo under test as the first extra_ref. Jobs that do not need to clone
anything will no-op.

Signed-off-by: Steve Kuznetsov skuznets@redhat.com

[stevekuznetsov]

@droslean @petr-muller @bbguimaraes PTAL

[petr-muller]

@stevekuznetsov I'll look tomorrow, but one thing that is surprising to me is that breaking-changes did not fail. Until now AFAIK prowgen (and determinizer) did not touch the files with periodics, which means that those jobs are are unorderded, sometimes contain falsy values that would not be serialized etc. I would expect enabling periodics with prowgen to reorder all of them.

[stevekuznetsov]

Very interesting. I'll look again.

[droslean]

Breaking-changes didn't fail, because no one is using the new feature in the existing ci-operator configs... Therefore, no new periodic jobs have been created, so no changes.

[stevekuznetsov]

From a quick skim, the determinizer does re-write the periodics, so I think Nikos is right here.

[droslean]

/lgtm
/hold

@petr-muller If you wanna take a look.

[petr-muller]

From a quick skim, the determinizer does re-write the periodics, so I think Nikos is right here.

I don't think so - the determinizer only does ReadFromDir and then WriteToDir and both of those only operate on Presubmits and Postsubmits...

See e.g. here, the keys are not sorted.

@droslean Please read what I wrote - I'm not expecting new jobs to be generated, but existing, non-generated periodics to be "determinized".

[petr-muller]

Ah, pruning:

ci-tools/cmd/ci-operator-prowgen/main.go

Line 609 in 3b3d772

if info.Type == "periodics" {

This explains why breaking-changes did not fail - because we continue to not touch periodics during pruning. This also causes the generated job to have ci-operator.openshift.io/prowgen-controlled: newly-generated label (newly-generated is the temporary value which pruning uses to recognize jobs that are not stale). So this should be fixed.

A good test would also be trying to generate a periodic for a component which already has some handcrafted periodics (like openshift/release) and checking if these handcrafted jobs are kept, determinized, and new ones correctly added.

ReadFromDir needs to be extended, too (that would make determinizer handle periodics)

[petr-muller]

Places that should taught about periodics:

• main.go:prune()
• main.go:pruneStaleJobs()
• possibly pkg/jobconfig/files.go: we are currently placing all *-periodics.yaml into the job-config-misc shard because they weren't previously tied to any branch, but now they will. This could be fixed for consistency, I believe nothing would break if we keep it that way.
• pkg/jobconfig/files.go:mergeConfigs currently only merges presubmits and postsubmits. Fixing this should make determinizer process periodics.

[stevekuznetsov]

/test breaking-changes

[hongkailiu]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: droslean, hongkailiu, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [droslean,hongkailiu,stevekuznetsov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[stevekuznetsov]

/hold cancel