pj-rehearse: support handcrafted jobs with UNRESOLVED_CONFIG

[petr-muller]

/hold
Builds on #1429

This is a thing of beauty! The tests needed to be completely rewritten (there were no real tests for the inlining functionality not working over the CM references -- removed in #1249 ), and the method was refactored - it is probably easier to review as new code than looking at the diff.

There is an annoying gotcha in that our code avoids resolving whole ci-op configs, but it resolves only the test relevant for the job, which is impossible to detect for handcrafted jobs in other way than looking at the --target=XXX parameter. Other option was to implement a "if no matching test was found, resolve all present, but that changes the behavior more significantly, and breaks multiple integration tests (which are a pinnacle of engineering themselves ) so I decided to go for the more targeted approach of inspecting --target.

Fixes: DPTP-1685

[petr-muller]

/hold cancel

[AlexNPavel]

We have a lot of jobs that use environment variables in the target and/or the UNRESOLVED_CONFIG itself. Could we run a quick substitute over the target and config using the env vars set for the container? Here is an example of a job that does this: https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/release/openshift-release-release-4.7-periodics.yaml#L5640:L5687

Both the BRANCH and CLUSTER_TYPE are defined as env vars in the container spec, so it should be as simple as running a for loop over those env vars and substituting all $(var_name) with the corresponding value.

[petr-muller]

Both the BRANCH and CLUSTER_TYPE are defined as env vars in the container spec, so it should be as simple as running a for loop over those env vars and substituting all $(var_name) with the corresponding value.

@AlexNPavel I was thinking about how to handle this best and I'd like to avoid getting into some weird middle state where we would substitute variables "somewhat like kube but not quite". E.g. then we'd need to handle also cases where something is first e.g. substituted in $CLUSTER_TYPE and such resolved value is then subsituted in UNRESOLVED_CONFIG? Maybe there is code we could reuse that does this for pods in kube? Not sure. But I'd like to tackle this in a dedicated followup.

[petr-muller]

@stevekuznetsov addressed, PTAL

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [petr-muller,stevekuznetsov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment