Skip to content

ARO-27360: Add ACR pull identity configured metric - #8840

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
twolff-gh:ARO-27360-acr-pull-metrics
Jul 20, 2026
Merged

ARO-27360: Add ACR pull identity configured metric#8840
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
twolff-gh:ARO-27360-acr-pull-metrics

Conversation

@twolff-gh

@twolff-gh twolff-gh commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds a hypershift_cluster_acr_pull_identity_configured gauge to the hostedcluster metrics collector. Emits 1 when an Azure HostedCluster has containerRegistry.credentials.managedIdentity.resourceID set, 0 when unset. Only emitted for Azure platform clusters.

This metric supports observability for the ACR pull via managed identity feature (OCPSTRAT-2951), letting SRE dashboards track adoption and alert on misconfiguration.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/ARO-27360

Main feature https://issues.redhat.com/browse/ARO-24037

Special notes for your reviewer:

CPO-side metric was evaluated and skipped — configmetrics runs in the guest cluster context without access to the HostedCluster/HostedControlPlane CR. The HO gauge covers it because the relationship is deterministic: if the HC has containerRegistry set, CPO writes userAssignedIdentityID into cloud.conf.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • New Features
    • Added a new per-cluster metric for Azure HostedClusters to indicate whether an ACR pull managed identity is configured.
    • The metric is only reported when Azure platform settings are present, and it reflects configured vs. not configured state.
  • Tests
    • Added coverage for the new metric across non-Azure, unset Azure, unconfigured, and configured cases.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 25, 2026
@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Adds a new per-HostedCluster Prometheus gauge named hypershift_cluster_acr_pull_identity_configured. The metric is emitted only for Azure platform HostedClusters and reports 1 when the Azure container registry managed identity ResourceID is set, otherwise 0. The per-cluster metrics path now calls the new collector, and a unit test covers absent and present cases.

Sequence Diagram(s)

sequenceDiagram
  participant HostedClusterMetrics as HostedCluster metrics collector
  participant AzureConfig as HostedCluster Azure config
  participant Prometheus as Prometheus gauge
  HostedClusterMetrics->>AzureConfig: read ContainerRegistry.Credentials.ManagedIdentity.ResourceID
  AzureConfig-->>HostedClusterMetrics: nil or configured value
  HostedClusterMetrics->>Prometheus: emit hypershift_cluster_acr_pull_identity_configured
Loading
🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The added test cases use static table-driven names; I found no timestamps, UUIDs, generated suffixes, node/namespace/IP values, or other run-to-run-changing titles.
Test Structure And Quality ✅ Passed PASS: The new table-driven unit test checks one metric behavior per case, uses the shared fake-client helper, and has no cleanup/timeout issues; no Ginkgo-style quality problems found.
Topology-Aware Scheduling Compatibility ✅ Passed Only Prometheus metric/test changes in hostedcluster metrics; no manifests, replicas, node selectors, affinity, or topology logic were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: This PR only adds unit tests; no Ginkgo e2e tests, IPv4-only literals, or external connectivity requirements were found.
No-Weak-Crypto ✅ Passed The added metric only checks an Azure ResourceID for emptiness; the touched files contain no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons.
Container-Privileges ✅ Passed PR only changes Go metrics collector/test code; no K8s manifests or privileged/host* / SYS_ADMIN / allowPrivilegeEscalation settings were added.
No-Sensitive-Data-In-Logs ✅ Passed The patch adds only metric collection and tests; I found no new log statements or sensitive values being logged in the changed files.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change by adding the ACR pull identity configured metric.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Jun 25, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
hypershift-operator/controllers/hostedcluster/metrics/metrics_test.go (1)

1319-1319: 📐 Maintainability & Code Quality | 🔵 Trivial

Use dto.MetricType_GAUGE.Enum() here instead of dto.MetricType(1). It avoids depending on the numeric ordinal and makes the intent clearer. Also applies to line 1351.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hypershift-operator/controllers/hostedcluster/metrics/metrics_test.go` at
line 1319, Update the metric type setup in metrics_test.go to use the enum
helper instead of hardcoded ordinals: replace the dto.MetricType(1) construction
with dto.MetricType_GAUGE.Enum() in both affected test cases. Locate the same
pattern in the metrics test helper/fixtures around the metric DTO initialization
and apply the same change wherever the gauge type is being set.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hypershift-operator/controllers/hostedcluster/metrics/metrics_test.go`:
- Around line 1296-1361: The test case names in metrics_test.go do not follow
the required “When ... it should ...” convention. Rename the affected cases in
the table-driven test around AcrPullIdentityConfiguredMetricName to descriptive
phrases like “When Azure has no containerRegistry configured it should emit 0”
and “When Azure has a containerRegistry configured it should emit 1”, keeping
the same intent but matching the coding guideline.

---

Nitpick comments:
In `@hypershift-operator/controllers/hostedcluster/metrics/metrics_test.go`:
- Line 1319: Update the metric type setup in metrics_test.go to use the enum
helper instead of hardcoded ordinals: replace the dto.MetricType(1) construction
with dto.MetricType_GAUGE.Enum() in both affected test cases. Locate the same
pattern in the metrics test helper/fixtures around the metric DTO initialization
and apply the same change wherever the gauge type is being set.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: c1b55b1b-57bf-4986-a8a8-1e2bb740f27c

📥 Commits

Reviewing files that changed from the base of the PR and between 334535c and 1d0289e.

📒 Files selected for processing (2)
  • hypershift-operator/controllers/hostedcluster/metrics/metrics.go
  • hypershift-operator/controllers/hostedcluster/metrics/metrics_test.go

Comment thread hypershift-operator/controllers/hostedcluster/metrics/metrics_test.go Outdated
@codecov

codecov Bot commented Jun 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.46%. Comparing base (dda6055) to head (be7bd91).
⚠️ Report is 153 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8840   +/-   ##
=======================================
  Coverage   43.45%   43.46%           
=======================================
  Files         771      771           
  Lines       95718    95733   +15     
=======================================
+ Hits        41597    41612   +15     
  Misses      51234    51234           
  Partials     2887     2887           
Files with missing lines Coverage Δ
...rator/controllers/hostedcluster/metrics/metrics.go 93.03% <100.00%> (+0.30%) ⬆️
Flag Coverage Δ
cmd-support 37.12% <ø> (ø)
cpo-hostedcontrolplane 45.21% <ø> (ø)
cpo-other 45.10% <ø> (ø)
hypershift-operator 53.68% <100.00%> (+0.02%) ⬆️
other 32.08% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@twolff-gh
twolff-gh force-pushed the ARO-27360-acr-pull-metrics branch 2 times, most recently from 6939414 to 1d0289e Compare July 6, 2026 23:06
@twolff-gh
twolff-gh marked this pull request as ready for review July 6, 2026 23:19
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 6, 2026
@openshift-ci
openshift-ci Bot requested review from clebs and sjenning July 6, 2026 23:19
@twolff-gh
twolff-gh force-pushed the ARO-27360-acr-pull-metrics branch from 1d0289e to 6a134b0 Compare July 7, 2026 17:31
@twolff-gh twolff-gh changed the title Add ACR pull identity configured metric (ARO-27360) ARO-27360: Add ACR pull identity configured metric Jul 7, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 7, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 7, 2026

Copy link
Copy Markdown

@twolff-gh: This pull request references ARO-27360 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Adds a hypershift_cluster_acr_pull_identity_configured gauge to the hostedcluster metrics collector. Emits 1 when an Azure HostedCluster has containerRegistry.credentials.managedIdentity.resourceID set, 0 when unset. Only emitted for Azure platform clusters.

This metric supports observability for the ACR pull via managed identity feature (OCPSTRAT-2951), letting SRE dashboards track adoption and alert on misconfiguration.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/ARO-27360

Special notes for your reviewer:

CPO-side metric was evaluated and skipped — configmetrics runs in the guest cluster context without access to the HostedCluster/HostedControlPlane CR. The HO gauge covers it because the relationship is deterministic: if the HC has containerRegistry set, CPO writes userAssignedIdentityID into cloud.conf.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • New Features
  • Added a new per-cluster metric for Azure HostedClusters to indicate whether an ACR pull managed identity is configured.
  • The metric is only reported when Azure platform settings are present, and it reflects configured vs. not configured state.
  • Tests
  • Added coverage for the new metric across non-Azure, unset Azure, unconfigured, and configured cases.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Add hypershift_cluster_acr_pull_identity_configured gauge metric that
indicates whether a HostedCluster has an ACR pull managed identity
configured. Emitted only for Azure platform clusters; value is 1 when a
ManagedIdentity ResourceID is set, 0 otherwise.

Signed-off-by: Todd Wolff <twolff@redhat.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
@twolff-gh
twolff-gh force-pushed the ARO-27360-acr-pull-metrics branch from 6a134b0 to be7bd91 Compare July 7, 2026 18:13
@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

I have all the information needed. Here is the analysis:

Test Failure Analysis Complete

Job Information

  • Prow Job: hypershift-cli-mce-50-on-pull-request (Red Hat Konflux pipeline)
  • Build ID: hypershift-cli-mce-50-on-pull-request-krljv
  • PR: ARO-27360: Add ACR pull identity configured metric #8840ARO-27360: Add ACR pull identity configured metric
  • Pipeline Started: 2026-07-07T18:13:40Z
  • Pipeline Completed: 2026-07-07T18:50:35Z
  • Failed Task: sast-snyk-check (duration: 6 seconds)
  • All Other Tasks (16/17): Succeeded ✅

Test Failure Analysis

Error

task sast-snyk-check has the status "TaskRunImagePullFailed":
the step "fetch-extra-artifacts" in TaskRun "hypershift-cli-mce-50-on-pull-request-krljv-sast-snyk-check"
failed to pull the image "quay.io/konflux-ci/build-trusted-artifacts:latest@sha256:83e9875f45cefd335641973a74db8bb4987cc98e47024aa4822eafd4150c315d".

The pod errored with the message: "Back-off pulling image: ErrImagePull: copying system image from manifest list:
parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable."

Summary

The sast-snyk-check task failed due to a transient 503 Service Unavailable error from quay.io when the Konflux pipeline tried to pull the container image quay.io/konflux-ci/build-trusted-artifacts:latest. This is an infrastructure-level failure in the Quay.io container registry — completely unrelated to the PR's code changes (ACR pull identity metric). All 16 other pipeline tasks (including the build, clone, security scans, and image index steps) succeeded. The failure occurred at the image-pull stage before any code analysis could even begin, meaning Snyk never ran.

Root Cause

The root cause is a transient Quay.io registry outage returning HTTP 503 during blob fetch:

  1. The Konflux sast-snyk-check task requires the helper image quay.io/konflux-ci/build-trusted-artifacts:latest@sha256:83e9875... for its fetch-extra-artifacts step.
  2. When Kubernetes attempted to pull this image, the Quay.io registry returned 503 Service Unavailable during the blob download phase (fetching blob: received unexpected HTTP status: 503 Service Unavailable).
  3. After the initial pull failure (ErrImagePull), Kubernetes entered exponential back-off (Back-off pulling image), and the task was marked as TaskRunImagePullFailed after exhausting retries.
  4. This is a Quay.io infrastructure issue — the image reference, digest, and manifest are all valid. The registry was temporarily unable to serve the blob. This type of 503 is a known transient failure pattern for container registries under load.
  5. No code from PR ARO-27360: Add ACR pull identity configured metric #8840 is involved — the failure occurred before any source code analysis step executed. The PR changes (adding an ACR pull identity configured metric) are irrelevant to this failure.
Recommendations
  1. Re-trigger the pipeline — This is a transient infrastructure failure. Simply re-running the Konflux pipeline (/retest or re-triggering via the Konflux UI) should resolve it, as Quay.io 503 errors are typically short-lived.
  2. No code changes needed — The PR's code changes are not related to this failure in any way.
  3. If failure persists on retry — Check Quay.io status page for ongoing incidents. Persistent 503s from quay.io/konflux-ci/ images would indicate a Konflux infrastructure issue to report to the Konflux team.
Evidence
Evidence Detail
Failed Task sast-snyk-check — 1 of 17 tasks failed; all others succeeded
Task Status TaskRunImagePullFailed — failure at container image pull, before any code analysis ran
Failed Step fetch-extra-artifacts — helper step to fetch trusted build artifacts
Failed Image quay.io/konflux-ci/build-trusted-artifacts:latest@sha256:83e9875f45cefd335641973a74db8bb4987cc98e47024aa4822eafd4150c315d
Error Type ErrImagePullBack-off pulling image (Kubernetes image pull back-off)
HTTP Error 503 Service Unavailable from Quay.io during blob fetch
Error Phase copying system image from manifest list: parsing image configuration: fetching blob
PR Relevance None — PR adds ACR pull identity metric; failure is in Konflux CI infrastructure
Pipeline Run hypershift-cli-mce-50-on-pull-request-krljv
Duration Task failed in 6 seconds (immediate image pull failure)

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/retest

@enxebre

enxebre commented Jul 9, 2026

Copy link
Copy Markdown
Member

/approve

@openshift-ci

openshift-ci Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre, twolff-gh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 9, 2026
@twolff-gh
twolff-gh requested a review from tmstff July 14, 2026 22:15
@mgencur

mgencur commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 16, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@tmstff

tmstff commented Jul 16, 2026

Copy link
Copy Markdown

/lgtm

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@tmstff: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/retest-required

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/test e2e-v2-gke

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/test e2e-aks

5 similar comments
@twolff-gh

Copy link
Copy Markdown
Contributor Author

/test e2e-aks

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/test e2e-aks

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/test e2e-aks

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/test e2e-aks

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/test e2e-aks

@twolff-gh

Copy link
Copy Markdown
Contributor Author

/verified by @twolff-gh

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 20, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@twolff-gh: This PR has been marked as verified by @twolff-gh.

Details

In response to this:

/verified by @twolff-gh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@twolff-gh: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit efffab7 into openshift:main Jul 20, 2026
41 checks passed
twolff-gh added a commit to twolff-gh/hypershift that referenced this pull request Jul 21, 2026
Backport of be7bd91 from main (PR openshift#8840), adapted for release-4.22
metrics file structure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
twolff-gh added a commit to twolff-gh/hypershift that referenced this pull request Jul 23, 2026
Backport of be7bd91 from main (PR openshift#8840), adapted for release-4.22
metrics file structure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants