[OCPNODE-553] Add CRD ImageDigestMirrorSet,ImageTagMirrorSet

[QiWang19]

- What I did

• Add support for ImageDigestMirrorSet,ImageTagMirrorSet mirror configuration CRDs.
• Reject the update of one of current icsp and new idms.itms resources if both exist to avoid having both icsp and new CRs on the same cluster.

Close card: https://issues.redhat.com/browse/OCPNODE-553
Epic: https://issues.redhat.com/browse/OCPNODE-521
Dependent PR: openshift/runtime-utils#15

- How to verify it

Apply following CR, check the /etc/containers/registries.conf

apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
  name: tag-mirror
spec:
  imageTagMirrors:
  - mirrors:
    - example.io/example/ubi-minimal 
    source: registry.access.redhat.com/ubi8/ubi-minimal
  - mirrors:
    - example.io/example/ubi-minimal 
    source: registry.access.redhat.com/ubi8/ubi-minimal-1
    mirrorSourcePolicy: NeverContactSource

/etc/containers/registries.conf

unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
short-name-mode = ""

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/ubi8/ubi-minimal"

  [[registry.mirror]]
    location = "example.io/example/ubi-minimal"
    pull-from-mirror = "tag-only"

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/ubi8/ubi-minimal-1"
  blocked = true

  [[registry.mirror]]
    location = "example.io/example/ubi-minimal"
    pull-from-mirror = "tag-only"

- Description for the changelog

Add support for ImageDigestMirrorSet,ImageDigestMirrorSet CRDs

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[mtrmac]

Just an initial pass (I didn’t actually read the added tests).

[nee1esh]

/retest

[QiWang19]

@yuqi-zhang @mtrmac This PR and dependent PR: openshift/runtime-utils#15 will update the registry.conf. And the runtime-utils PR replaced sysregistriesv2.Registry.MirrorByDigestOnly with new per-mirror field sysregistriesv2.Endpoint.PullFromMirror to edit the registry.conf. So I think this PR should also update isSafeContainerRegistryConfChanges().

machine-config-operator/pkg/daemon/drain.go

Lines 178 to 187 in be77042

	// isSafeContainerRegistryConfChanges looks inside old and new versions of registries.conf file.
	// It compares the content and determines whether changes made are safe or not. This will
	// help MCD to decide whether we can skip node drain for applied changes into container
	// registry.
	// Currently, we consider following container registry config changes as safe to skip node drain:
	// 1. A new mirror is added to an existing registry that has `mirror-by-digest-only=true`
	// 2. A new registry has been added that has `mirror-by-digest-only=true`
	// See https://bugzilla.redhat.com/show_bug.cgi?id=1943315
	//nolint:gocyclo
	func isSafeContainerRegistryConfChanges(oldIgnConfig, newIgnConfig ign3types.Config) (bool, error) {

Not sure how the safe conditions are defined, but looking at https://bugzilla.redhat.com/show_bug.cgi?id=1943315#c1, we could propose the following as safe?

// Currently, we consider following container registry config changes as safe to skip node drain:
// 1. A new mirror is added to an existing registry that has per-mirror setting `pull-from-mirror=digest-only` or registry level setting `mirror-by-digest-only=true`
// 2. A new registry has been added that has `mirror-by-digest-only=true` or per-mirror setting `pull-from-mirror=digest-only` for all of its mirrors

[openshift-ci]

@QiWang19: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mtrmac]

LGTM overall. (FWIW the PR doesn’t currently compile).

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 97ddbfe and 1 for PR HEAD 7971961 in total

[yuqi-zhang]

Only e2e-aws-ovn is required, is failing, and being retested, so skipping tests won't really help us either way. The other failures don't block the PR

Or did you mean you wanted me to override e2e-aws-ovn?

[QiWang19]

Yes, override e2e-aws-ovn. The e2e-aws-ovn was retested several times and failed. Can we override it?

[yuqi-zhang]

The latest commit did have a passing run: https://prow.ci.openshift.org/pr-history/?org=openshift&repo=machine-config-operator&pr=3037

So I think it's most likely ok. Still, I am uncomfortable overriding a required job unless the fix is urgent or directly attributed to it. Let's wait until the end of today, and, if there are no passes, I can look to override