OCPBUGS-99534: etcd initialization bugs after etcdctl snapshot restore - #9048
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
Skipping CI for Draft Pull Request. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe etcd StatefulSet now computes and reuses its initial cluster configuration. It injects pod and namespace identity into the restore init container. It places Sequence Diagram(s)sequenceDiagram
participant StatefulSetAdapter
participant EtcdInitContainer
participant SnapshotTool
participant EtcdData
StatefulSetAdapter->>EtcdInitContainer: Inject member and cluster identity
StatefulSetAdapter->>EtcdInitContainer: Start before reset-member
EtcdInitContainer->>SnapshotTool: Restore snapshot with explicit cluster settings
SnapshotTool->>EtcdData: Stage restored data in /var/lib/restore
Suggested reviewers: 🚥 Pre-merge checks | ✅ 10 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (10 passed)
Full details: Stable And Deterministic Test NamesExplanation PASS. The changed tests use static Full details: Test Structure And QualityExplanation The added tests use standalone Gomega assertions without meaningful failure messages. In Resolution Add a diagnostic message to every assertion introduced by this change. For example, identify the expected environment variable in each Full details: Topology-Aware Scheduling CompatibilityExplanation PASS — the changed implementation adds no topology-sensitive scheduling constraint. The diff only changes etcd snapshot-restore arguments, environment variables, and init-container ordering. The existing Full details: Ipv6 And Disconnected Network Test CompatibilityExplanation PASS: The pull request adds standard Go unit tests using Full details: No-Weak-CryptoExplanation No weak-crypto usage was introduced. The PR diff from the merge base changes etcd restore arguments, environment variables, init-container ordering, tests, and generated fixtures. Exact added-line searches found no MD5, SHA-1, DES, 3DES, RC4, Blowfish, or ECB usage, and no crypto APIs or constant-time comparison changes. The Full details: Container-PrivilegesExplanation No pull-request-introduced container privilege violation was found. The changed etcd source adds no Full details: No-Sensitive-Data-In-LogsExplanation PASS. The pull request adds no new logging of passwords, tokens, API keys, PII, or customer data. The changed script passes member and cluster values to ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go (1)
1142-1145: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse the required test-case description format.
Rename this case to
When etcd snapshot restore is configured, it should reconcile the expected components.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go` around lines 1142 - 1145, Rename the test case in the relevant test-case table from “Default feature set, etcd snapshot restore” to “When etcd snapshot restore is configured, it should reconcile the expected components,” without changing its featureSet, subDirSuffix, or test behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go`:
- Around line 1142-1145: Rename the test case in the relevant test-case table
from “Default feature set, etcd snapshot restore” to “When etcd snapshot restore
is configured, it should reconcile the expected components,” without changing
its featureSet, subDirSuffix, or test behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 32129925-6bcc-42aa-afcd-854baa0dae02
📒 Files selected for processing (4)
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.gocontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd-init.shcontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd_test.gocontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/statefulset.go
bee7b14 to
62eb783
Compare
62eb783 to
40ce74a
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9048 +/- ##
==========================================
+ Coverage 47.32% 47.33% +0.01%
==========================================
Files 792 792
Lines 99678 99697 +19
==========================================
+ Hits 47169 47195 +26
+ Misses 49347 49341 -6
+ Partials 3162 3161 -1
... and 1 file with indirect coverage changes
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@control-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd-init.sh`:
- Around line 26-40: Add fail-fast handling to the etcd-init script before the
restore branches, such as enabling errexit with set -e, so failures from either
the etcdutl or etcdctl snapshot restore commands stop execution. Ensure the
subsequent promotion of /var/lib/restore into /var/lib/data cannot run after a
partial or failed restore.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 7464c0d5-025d-49c5-b037-2d3b3165646d
📒 Files selected for processing (4)
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.gocontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd-init.shcontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd_test.gocontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/statefulset.go
🚧 Files skipped from review as they are similar to previous changes (3)
- control-plane-operator/controllers/hostedcontrolplane/v2/etcd/statefulset.go
- control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
- control-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd_test.go
| etcdutl snapshot restore /tmp/snapshot \ | ||
| --data-dir=/var/lib/restore \ | ||
| --name "${HOSTNAME}" \ | ||
| --initial-advertise-peer-urls "${PEER_URL}" \ | ||
| --initial-cluster "${ETCD_INITIAL_CLUSTER}" \ | ||
| --initial-cluster-token "${HCP_NAMESPACE}" | ||
| elif [ -x /usr/bin/etcdctl ]; then | ||
| echo "INFO: using etcdctl (etcd 3.5.x)" | ||
| env ETCDCTL_API=3 /usr/bin/etcdctl -w table snapshot status /tmp/snapshot | ||
| env ETCDCTL_API=3 /usr/bin/etcdctl snapshot restore /tmp/snapshot --data-dir=/var/lib/restore | ||
| env ETCDCTL_API=3 /usr/bin/etcdctl snapshot restore /tmp/snapshot \ | ||
| --data-dir=/var/lib/restore \ | ||
| --name "${HOSTNAME}" \ | ||
| --initial-advertise-peer-urls "${PEER_URL}" \ | ||
| --initial-cluster "${ETCD_INITIAL_CLUSTER}" \ | ||
| --initial-cluster-token "${HCP_NAMESPACE}" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Do not promote a failed or partial restore.
This script does not enable errexit or check either restore command’s exit status. If etcdutl or etcdctl creates partial contents and then fails, Lines [46-47] still move that incomplete database into /var/lib/data, potentially causing etcd corruption or a CrashLoopBackOff. Add set -e near the top or explicitly abort when either restore command fails.
Proposed minimal fix
+set -euo pipefail
+
# If /var/lib/data is not empty we exit early, since this means an etcd database already exists📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| etcdutl snapshot restore /tmp/snapshot \ | |
| --data-dir=/var/lib/restore \ | |
| --name "${HOSTNAME}" \ | |
| --initial-advertise-peer-urls "${PEER_URL}" \ | |
| --initial-cluster "${ETCD_INITIAL_CLUSTER}" \ | |
| --initial-cluster-token "${HCP_NAMESPACE}" | |
| elif [ -x /usr/bin/etcdctl ]; then | |
| echo "INFO: using etcdctl (etcd 3.5.x)" | |
| env ETCDCTL_API=3 /usr/bin/etcdctl -w table snapshot status /tmp/snapshot | |
| env ETCDCTL_API=3 /usr/bin/etcdctl snapshot restore /tmp/snapshot --data-dir=/var/lib/restore | |
| env ETCDCTL_API=3 /usr/bin/etcdctl snapshot restore /tmp/snapshot \ | |
| --data-dir=/var/lib/restore \ | |
| --name "${HOSTNAME}" \ | |
| --initial-advertise-peer-urls "${PEER_URL}" \ | |
| --initial-cluster "${ETCD_INITIAL_CLUSTER}" \ | |
| --initial-cluster-token "${HCP_NAMESPACE}" | |
| set -euo pipefail | |
| etcdutl snapshot restore /tmp/snapshot \ | |
| --data-dir=/var/lib/restore \ | |
| --name "${HOSTNAME}" \ | |
| --initial-advertise-peer-urls "${PEER_URL}" \ | |
| --initial-cluster "${ETCD_INITIAL_CLUSTER}" \ | |
| --initial-cluster-token "${HCP_NAMESPACE}" | |
| elif [ -x /usr/bin/etcdctl ]; then | |
| echo "INFO: using etcdctl (etcd 3.5.x)" | |
| env ETCDCTL_API=3 /usr/bin/etcdctl -w table snapshot status /tmp/snapshot | |
| env ETCDCTL_API=3 /usr/bin/etcdctl snapshot restore /tmp/snapshot \ | |
| --data-dir=/var/lib/restore \ | |
| --name "${HOSTNAME}" \ | |
| --initial-advertise-peer-urls "${PEER_URL}" \ | |
| --initial-cluster "${ETCD_INITIAL_CLUSTER}" \ | |
| --initial-cluster-token "${HCP_NAMESPACE}" |
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 33-33: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/snapshot
Note: [CWE-377] Insecure Temporary File.
(predictable-tmp-file-bash)
[warning] 34-34: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/snapshot
Note: [CWE-377] Insecure Temporary File.
(predictable-tmp-file-bash)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@control-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd-init.sh`
around lines 26 - 40, Add fail-fast handling to the etcd-init script before the
restore branches, such as enabling errexit with set -e, so failures from either
the etcdutl or etcdctl snapshot restore commands stop execution. Ensure the
subsequent promotion of /var/lib/restore into /var/lib/data cannot run after a
partial or failed restore.
c77b637 to
859b305
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (1)
control-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd-init.sh (1)
26-40: 🩺 Stability & Availability | 🟠 MajorDo not promote a failed or partial restore.
The previous finding remains valid: if
etcdutloretcdctlfails, execution can still reach Line [46] and move incomplete contents into/var/lib/data. Add fail-fast handling or explicit checks around both restore commands before promotion.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@control-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd-init.sh` around lines 26 - 40, Update the restore branches in the etcdutl/etcdctl selection to fail immediately when either snapshot restore command fails, before execution can reach the promotion that moves contents into /var/lib/data. Enable fail-fast behavior or explicitly check each restore command’s exit status, while preserving the existing status inspection and restore arguments.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@control-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd-init.sh`:
- Around line 26-40: Update the restore branches in the etcdutl/etcdctl
selection to fail immediately when either snapshot restore command fails, before
execution can reach the promotion that moves contents into /var/lib/data. Enable
fail-fast behavior or explicitly check each restore command’s exit status, while
preserving the existing status inspection and restore arguments.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 73319c35-161d-469e-9fa1-cfff9f4fbdc2
⛔ Files ignored due to path filters (240)
control-plane-operator/controllers/hostedcontrolplane/testdata/aws-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_aws_cloud_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/aws-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_aws_cloud_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/aws-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_aws_cloud_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/aws-node-termination-handler/EtcdRestore/zz_fixture_TestControlPlaneComponents_aws_node_termination_handler_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/aws-node-termination-handler/EtcdRestore/zz_fixture_TestControlPlaneComponents_aws_node_termination_handler_creds_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/aws-node-termination-handler/EtcdRestore/zz_fixture_TestControlPlaneComponents_aws_node_termination_handler_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/azure-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_azure_cloud_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/azure-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_azure_cloud_config_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/azure-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_azure_cloud_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/azure-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_azure_cloud_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/azure-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_azure_cloud_controller_manager_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/catalog-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_catalog_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/catalog-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_catalog_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/catalog-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_catalog_operator_metrics_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/catalog-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_catalog_operator_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/certified-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_certified_operators_catalog_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/certified-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_certified_operators_catalog_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/certified-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_certified_operators_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cloud-credential-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cloud_credential_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cloud-credential-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cloud_credential_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cloud-credential-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cloud_credential_operator_service_account_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-autoscaler/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_autoscaler_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-autoscaler/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_autoscaler_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-autoscaler/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_autoscaler_podmonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-autoscaler/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_autoscaler_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-autoscaler/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_autoscaler_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-autoscaler/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_autoscaler_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-image-registry-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_image_registry_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-image-registry-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_image_registry_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-image-registry-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_image_registry_operator_podmonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-image-registry-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_image_registry_controller_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-network-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_network_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-network-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_network_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-network-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_network_operator_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-network-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_network_operator_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-network-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_network_operator_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-node-tuning-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_node_tuning_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-node-tuning-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_node_tuning_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-node-tuning-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_node_tuning_operator_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-node-tuning-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_node_tuning_operator_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-node-tuning-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_node_tuning_operator_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-node-tuning-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_node_tuning_operator_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-node-tuning-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_node_tuning_operator_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-policy-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_policy_controller_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-policy-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_policy_controller_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-policy-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_policy_controller_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-storage-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_storage_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-storage-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_storage_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-storage-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_storage_operator_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-storage-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_storage_operator_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-storage-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_storage_operator_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_version_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_version_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_version_operator_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_cluster_version_operator_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/community-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_community_operators_catalog_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/community-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_community_operators_catalog_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/community-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_community_operators_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/csi-snapshot-controller-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_csi_snapshot_controller_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/csi-snapshot-controller-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_csi_snapshot_controller_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/csi-snapshot-controller-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_csi_snapshot_controller_operator_role_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/csi-snapshot-controller-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_csi_snapshot_controller_operator_role_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/csi-snapshot-controller-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_csi_snapshot_controller_operator_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/dns-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_dns_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/dns-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_dns_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/dns-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_dns_operator_service_account_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/endpoint-resolver/EtcdRestore/zz_fixture_TestControlPlaneComponents_endpoint_resolver_ca_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/endpoint-resolver/EtcdRestore/zz_fixture_TestControlPlaneComponents_endpoint_resolver_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/endpoint-resolver/EtcdRestore/zz_fixture_TestControlPlaneComponents_endpoint_resolver_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/endpoint-resolver/EtcdRestore/zz_fixture_TestControlPlaneComponents_endpoint_resolver_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/endpoint-resolver/EtcdRestore/zz_fixture_TestControlPlaneComponents_endpoint_resolver_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/endpoint-resolver/EtcdRestore/zz_fixture_TestControlPlaneComponents_endpoint_resolver_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/endpoint-resolver/EtcdRestore/zz_fixture_TestControlPlaneComponents_endpoint_resolver_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/endpoint-resolver/EtcdRestore/zz_fixture_TestControlPlaneComponents_endpoint_resolver_serving_cert_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_client_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_discovery_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_poddisruptionbudget.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_self_register_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_self_register_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/etcd/EtcdRestore/zz_fixture_TestControlPlaneComponents_etcd_statefulset.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/featuregate-generator/EtcdRestore/zz_fixture_TestControlPlaneComponents_featuregate_generator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/featuregate-generator/EtcdRestore/zz_fixture_TestControlPlaneComponents_featuregate_generator_job.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/gcp-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_gcp_cloud_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/gcp-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_gcp_cloud_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/gcp-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_gcp_cloud_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/hosted-cluster-config-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_hosted_cluster_config_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/hosted-cluster-config-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_hosted_cluster_config_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/hosted-cluster-config-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_hosted_cluster_config_operator_podmonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/hosted-cluster-config-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_hosted_cluster_config_operator_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/hosted-cluster-config-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_hosted_cluster_config_operator_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/hosted-cluster-config-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_hosted_cluster_config_operator_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_proxy_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_proxy_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_proxy_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_ca_cert_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_podmonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_route.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ignition-server/EtcdRestore/zz_fixture_TestControlPlaneComponents_ignition_server_serving_cert_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_ingress_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_ingress_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_ingress_operator_podmonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_ingress_operator_service_account_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/konnectivity-agent/EtcdRestore/zz_fixture_TestControlPlaneComponents_konnectivity_agent_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/konnectivity-agent/EtcdRestore/zz_fixture_TestControlPlaneComponents_konnectivity_agent_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/konnectivity-agent/EtcdRestore/zz_fixture_TestControlPlaneComponents_konnectivity_agent_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_admin_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_auth_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_aws_pod_identity_webhook_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_bootstrap_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_hcco_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kas_audit_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kas_authentication_token_webhook_config_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kas_bootstrap_container_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kas_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kas_egress_selector_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_konnectivity_server_local_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_apiserver_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_apiserver_poddisruptionbudget.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_apiserver_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_localhost_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_metadata_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_recording_rules_prometheusrule.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_service_network_admin_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kcm_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_controller_manager_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_controller_manager_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_controller_manager_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_recycler_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_scheduler_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_scheduler_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_scheduler_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_scheduler_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_scheduler_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_scheduler_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-storage-version-migrator/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_storage_version_migrator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-storage-version-migrator/EtcdRestore/zz_fixture_TestControlPlaneComponents_kube_storage_version_migrator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_cloud_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_cloud_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_cloud_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_cloud_controller_manager_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_cloud_controller_manager_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_cloud_controller_manager_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-csi-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_driver_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-csi-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_csi_controller_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-csi-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_csi_controller_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-csi-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_csi_controller_service_account_kubeconfig_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-csi-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_csi_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-csi-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_csi_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kubevirt-csi-controller/EtcdRestore/zz_fixture_TestControlPlaneComponents_kubevirt_csi_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/machine-approver/EtcdRestore/zz_fixture_TestControlPlaneComponents_machine_approver_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/machine-approver/EtcdRestore/zz_fixture_TestControlPlaneComponents_machine_approver_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/machine-approver/EtcdRestore/zz_fixture_TestControlPlaneComponents_machine_approver_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/machine-approver/EtcdRestore/zz_fixture_TestControlPlaneComponents_machine_approver_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/machine-approver/EtcdRestore/zz_fixture_TestControlPlaneComponents_machine_approver_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/machine-approver/EtcdRestore/zz_fixture_TestControlPlaneComponents_machine_approver_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/metrics-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_metrics_proxy_ca_cert_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/metrics-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_metrics_proxy_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/metrics-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_metrics_proxy_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/metrics-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_metrics_proxy_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/metrics-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_metrics_proxy_route.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/metrics-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_metrics_proxy_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/metrics-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_metrics_proxy_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/metrics-proxy/EtcdRestore/zz_fixture_TestControlPlaneComponents_metrics_proxy_serving_cert_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_audit_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_default_error_template_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_default_login_template_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_default_provider_selection_template_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_poddisruptionbudget.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/oauth-openshift/EtcdRestore/zz_fixture_TestControlPlaneComponents_oauth_openshift_session_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-collect-profiles/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_collect_profiles_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-collect-profiles/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_collect_profiles_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-collect-profiles/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_collect_profiles_cronjob.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-collect-profiles/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_collect_profiles_role.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-collect-profiles/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_collect_profiles_rolebinding.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-collect-profiles/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_collect_profiles_serviceaccount.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-collect-profiles/EtcdRestore/zz_fixture_TestControlPlaneComponents_pprof_cert_secret.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_operator_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_operator_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_operator_metrics_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/olm-operator/EtcdRestore/zz_fixture_TestControlPlaneComponents_olm_operator_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_apiserver_audit_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_apiserver_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_apiserver_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_apiserver_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_apiserver_poddisruptionbudget.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_apiserver_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_controller_manager_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_controller_manager_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_controller_manager_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_audit_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_poddisruptionbudget.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-route-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_route_controller_manager_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-route-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_route_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-route-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_route_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-route-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_route_controller_manager_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-route-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openshift_route_controller_manager_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openstack-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openstack_cloud_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openstack-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openstack_cloud_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openstack-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openstack_cloud_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/openstack-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_openstack_trusted_ca_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/packageserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_packageserver_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/packageserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_packageserver_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/packageserver/EtcdRestore/zz_fixture_TestControlPlaneComponents_packageserver_pdb_poddisruptionbudget.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/powervs-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_ccm_config_configmap.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/powervs-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_powervs_cloud_controller_manager_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/powervs-cloud-controller-manager/EtcdRestore/zz_fixture_TestControlPlaneComponents_powervs_cloud_controller_manager_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/redhat-marketplace-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_redhat_marketplace_catalog_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/redhat-marketplace-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_redhat_marketplace_catalog_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/redhat-marketplace-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_redhat_marketplace_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/redhat-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_catalogs_imagestream.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/redhat-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_redhat_operators_catalog_controlplanecomponent.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/redhat-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_redhat_operators_catalog_deployment.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/redhat-operators-catalog/EtcdRestore/zz_fixture_TestControlPlaneComponents_redhat_operators_service.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/router/EtcdRestore/zz_fixture_TestControlPlaneComponents_router_controlplanecomponent.yamlis excluded by!**/testdata/**
📒 Files selected for processing (4)
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.gocontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd-init.shcontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd_test.gocontrol-plane-operator/controllers/hostedcontrolplane/v2/etcd/statefulset.go
🚧 Files skipped from review as they are similar to previous changes (3)
- control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
- control-plane-operator/controllers/hostedcontrolplane/v2/etcd/statefulset.go
- control-plane-operator/controllers/hostedcontrolplane/v2/etcd/etcd_test.go
|
@tony-schndr is all the 16k lines of changes really needed to fix this bug? Could you please use the restructure commit skill in the hypershift repo to split up the changes into appropriate commits?; that would make it much easier to review. |
|
/retitle OCPBUGS-99534: etcd initialization bugs after etcdctl snapshot restore |
|
/test unit |
|
/hold cancel |
|
/hold Revision 5f82787 was retested 3 times: holding |
|
/test unit |
|
looks like unit is failing consistently. @tony-schndr do you mind taking a look? |
Fixes OCPBUGS-99310 Fix etcd split brain caused by single member initialization. - Previously etcd members were started without information about other cluster members, causing each to initialize individually. Added --name, --initial-advertise-peer-urls, --initial-cluster, and --initial-cluster-token to etcdctl (and utl) snapshot restore to ensure members are properly initialized during restore. Fix member ID mismatch on snapshot restore by running etcd-init before reset-member. - Previously, reset-member ran first and, if the cluster was already up, would remove and re-add the local member with a new ID. etcd-init then restored the snapshot with the old member ID, causing a WAL/cluster membership mismatch and CrashLoopBackOff. Running etcd-init first ensures the data directory exists when reset-member runs, so it takes the no-op "member has data" path and leaves membership unchanged. Signed-off-by: Tony Schneider <tschneid@redhat.com>
Pass --bump-revision 1000000000 --mark-compacted to etcdutl and etcdctl snapshot restore invocations so that post-restore revisions are always greater than any revision the cluster has seen, and all watches are terminated on startup. Without this, informer caches may observe stale revisions from before the snapshot was taken. Ref: https://etcd.io/docs/v3.5/op-guide/recovery/ Signed-off-by: Tony Schneider <tschneid@redhat.com>
… tests Signed-off-by: Tony Schneider <tschneid@redhat.com>
5f82787 to
6625825
Compare
|
/hold cancel |
|
/lgtm |
|
Scheduling tests matching the |
|
/verified by @adalrsjr1 10 consecutive backup runs, all completed successfully and no split brain in etcd fleet. All etcd members see each other before backup and after restore. Being this just a rebase from main I will consider this as verified as @adalrsjr1 did some time ago: #9048 (comment) |
|
@jparrill: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hlipsig, jparrill, tony-schndr The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@tony-schndr: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@tony-schndr: Jira Issue Verification Checks: Jira Issue OCPBUGS-99534 Jira Issue OCPBUGS-99534 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
What this PR does / why we need it:
Fix etcd split brain caused by single member initialization.
Fix member ID mismatch on snapshot restore by running etcd-init before reset-member.
Adds
--bump-revision 1000000000 --mark-compactedflags to etcdctl|etcdutl snapshot restore to ensure etcd object revisions are always greater than any revision the cluster has seen. This ensures that all watches are terminated and etcd does not respond to requests about revisions that happened after taking the snapshot - effectively invalidating its informer caches. ref https://etcd.io/docs/v3.5/op-guide/recovery/Which issue(s) this PR fixes:
Fixes OCPBUGS-99534
Special notes for your reviewer:
Checklist:
Summary by CodeRabbit
Summary by CodeRabbit