IR-461: blocked-edges/4.14.*-AzureRegistryImagePreservation: Look for active registry use

[wking]

Some clusters do not excercise the internal image registry, and we do not want to bother those admins about this risk. Look at imageregistry_request_duration_seconds_count (which only records registry activity like BlobStore.Create) as a sign that the registry is actively used to handle images. Compare with
imageregistry_http_request_duration_seconds_count, which the registry serves even when it is not actively being used to manage images, as a sign that registry metrics are working, to distinguish between "we know the registry is not being used" and "registry metrics collection is broken, and we're blind to registry use".

In 4.14, the registry component became an optional component, but it is not optional in 4.13. And it's 4.13 clusters evaluating this PromQL because of the from: 4[.]13[.].*. So we don't have to worry about "what if imageregistry_http_request_duration_seconds_count is not available?" cases outside of the "registry metrics collection is broken" case.

The registry does not persist these *_count values across container restarts, so there is a window of exposure like:

1. Registry activity increments imageregistry_request_duration_seconds_count counters.
2. PromQL correctly identifies "this cluster is using its registry".
3. Registry container restarts, zeroing its *_count buckets.
4. If all registry containers restart before there is new registry activity, the imageregistry_request_duration_seconds_count signal may go away.
5. New registry activity repopulates imageregistry_request_duration_seconds_count, return to step 1.

The chances that all registry pods restart at the same time seems low, outside of:

• Updates or other registry-hosting-node reboots, where clusters that actively use the in-cluster registry are likely to have pods consuming images from that registry need re-pulling, which would drive fresh registry traffic and repopulate imageregistry_request_duration_seconds_count.
• Registry reconfiguration, which would not drive fresh registry traffic, but which seems like unlikely admin activity in a cluster where the internal-registry is not being used.

I'm also using the max_over_time with a 1 hour lookback to try to smear over small durations of "waiting for new registry activity to repopulate imageregistry_request_duration_seconds_count".

And now that the PromQL is getting pretty complicated, I'm also adjusting label management so it's easier to see from the result which clauses are matching for a particular cluster, in case we need to debug why we're matching when we don't expect to, or not matching when we do expect to.

I'm also adding _id="" placeholders for HyperShift, see 5cb2e93 (#3591) for more on that.

Generated by editing 4.14.1's by hand and copying out to the others with:

$ ls blocked-edges/*AzureRegistryImagePreservation.yaml | grep -v '/4[.]14[.]1-A' | while read X; do TO="$(grep '^to: ' "${X}")"; sed "s/^to: .*/${TO}/" blocked-edges/4.14.1-AzureRegistryImagePreservation.yaml > "${X}"; done

[openshift-ci-robot]

@wking: This pull request references IR-461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.16.0" version, but no target version was set.

Details

In response to this:

Some clusters do not excercise the internal image registry, and we do not want to bother those admins about this risk. Look at imageregistry_request_duration_seconds_count (which only records registry activity like BlobStore.Create) as a sign that the registry is actively used to handle images. Compare with
imageregistry_http_request_duration_seconds_count, which the registry serves even when it is not actively being used to manage images, as a sign that registry metrics are working, to distinguish between "we know the registry is not being used" and "registry metrics collection is broken, and we're blind to registry use".

In 4.14, the registry component became an optional component, but it is not optional in 4.13. And it's 4.13 clusters evaluating this PromQL because of the from: 4[.]13[.].*. So we don't have to worry about "what if imageregistry_http_request_duration_seconds_count is not available?" cases outside of the "registry metrics collection is broken" case.

The registry does not persist these *_count values across container restarts, so there is a window of exposure like:

1. Registry activity increments imageregistry_request_duration_seconds_count counters.
2. PromQL correctly identifies "this cluster is using its registry".
3. Registry container restarts, zeroing its *_count buckets.
4. If all registry containers restart before there is new registry activity, the imageregistry_request_duration_seconds_count signal may go away.
5. New registry activity repopulates imageregistry_request_duration_seconds_count, return to step 1.

The chances that all registry pods restart at the same time seems low, outside of:

• Updates or other registry-hosting-node reboots, where clusters that actively use the in-cluster registry are likely to have pods consuming images from that registry need re-pulling, which would drive fresh registry traffic and repopulate imageregistry_request_duration_seconds_count.
• Registry reconfiguration, which would not drive fresh registry traffic, but which seems like unlikely admin activity in a cluster where the internal-registry is not being used.

I'm also using the max_over_time with a 1 hour lookback to try to smear over small durations of "waiting for new registry activity to repopulate imageregistry_request_duration_seconds_count".

And now that the PromQL is getting pretty complicated, I'm also adjusting label management so it's easier to see from the result which clauses are matching for a particular cluster, in case we need to debug why we're matching when we don't expect to, or not matching when we do expect to.

I'm also adding _id="" placeholders for HyperShift, see 5cb2e93 (#3591) for more on that.

Generated by editing 4.14.1's by hand and copying out to the others with:

$ ls blocked-edges/*AzureRegistryImagePreservation.yaml | grep -v '/4[.]14[.]1-A' | while read X; do TO="$(grep '^to: ' "${X}")"; sed "s/^to: .*/${TO}/" blocked-edges/4.14.1-AzureRegistryImagePreservation.yaml > "${X}"; done

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[wking]

Testing the new PromQL in 4.13.33's azure-ovn-upgrade-4.13-micro run in PromeCIeus, it notices some registry use late in the update after the nodes rolled:

[wking]

And testing vs. 4.13.33's aws-sdn-upgrade-4.13-micro run in PromeCIeus, it sees both "no registry activity" and "detected registry activity", but knows it is not exposed in either case because it is AWS and not Azure:

[bennerv]

LGTM on aro:

[wking]

And refreshing my 4.13.33 screenshots, now that we've added the ARO condition, I get similar results to what I got before, and they recognize that they aren't ARO clusters:

[sdodson]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdodson, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment