Enhancement: Etcd sharding by resource kind for HyperShift by jhjaggars · Pull Request #1979 · openshift/enhancements

jhjaggars · 2026-04-22T16:10:27Z

Proposes etcd sharding by Kubernetes resource kind for HyperShift hosted control planes, enabling distribution of resources across multiple independent etcd deployments for improved scalability and performance.

Each etcd shard is registered as an independent ControlPlaneComponent within the CPO v2 component framework, inheriting all framework features automatically. KAS is configured with --etcd-servers-overrides to route resources to the appropriate shard.

Conditional component registration: shard components registered at CPO startup based on HCP spec; shard list is immutable after creation
Framework reuse: each shard uses NewStatefulSetComponent with WithAssetDir("etcd") and WithTemplateData, inheriting priority class, topology spread, scale-to-zero, PDB, etc.
Managed + Unmanaged: supports both managed (CPO-deployed) and unmanaged (external) etcd sharding
Feature-gated: EtcdSharding feature gate in TechPreviewNoUpgrade
API additions: EtcdShardResource, ManagedEtcdShardSpec, ManagedEtcdShardStorageSpec, EtcdShardSchedulingSpec, UnmanagedEtcdShardSpec added to hypershift.openshift.io/v1beta1

cc @enxebre @sjenning @csrwng

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Rename "v2 framework" to "CPO component framework" throughout - Scope hcp CLI references to self-hosted/MCE deployments - Replace EtcdLike interface with strings.HasPrefix prefix check - Replace per-manifest adapt functions with TemplatedProvider approach - Update API types to match kube-api-linter output (omitzero, value types, MinItems/MinLength bounds, omitempty on list map keys) - Drop data-policy annotation — backup determined by storage type (PVC = backed up, EmptyDir = not) - Add parent-level CEL rule preventing shard removal once configured - Add cross-shard duplicate resource prefix CEL validation - Add CEL rule preventing storageClassName on EmptyDir storage - Make scheduling mutable (no data migration needed for placement) - Clarify replicas override controllerAvailabilityPolicy when set - Document CPO restart idempotency for conditional registration - Make downgrade incompatibility for sharded HCPs explicit - Document wait-for-etcd extension mechanism for multi-shard - Document defrag controller sidecar behavior with multiple shards - Add ServiceMonitor and PDB to NewShardComponent registration - Fix stale DataPolicy reference in ManagedEtcdShardStorageSpec - Fix ResourcePrefixes godoc listing "/" as valid for non-default shards - Update Alternative C to reflect partial template adoption Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@JoelSpeed

- Replace resourcePrefixes []string with structured EtcdShardResource type (apiGroup + resource fields with proper validation) - Restructure ManagedEtcdShardStorageSpec as discriminated union with nested PersistentVolume spec and CEL union rule - Change shard name validation to DNS1123 label (was DNS1035-like) - Make storage mutable day-2 (switching PVC/EmptyDir doesn't need cluster recreation) - Make replicas non-pointer int32 (zero value is never valid) - Use CEL url library for endpoint validation (isURL + getScheme) - Prevent adding shards to unsharded clusters (has(oldSelf.shards) == has(self.shards)) - Move MinProperties=1 from Scheduling field to EtcdShardSchedulingSpec struct - Remove redundant omitempty on struct fields (omitzero only) - Add commented-out label key/value validation on nodeSelector (CEL cost budget) - Fix EmptyDir data loss description (survives container restarts) - Add single-replica failure mode note - Add shard rename envtest case - StorageClassName MaxLength 253, DNS1123 subdomain - Add @JoelSpeed to api-approvers - Update last-updated to 2026-04-30 All type changes verified against kube-api-linter (0 issues). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@JoelSpeed

- Remove +immutable tags (non-functional, replaced by CEL self == oldSelf) - Make APIGroup a *string for empty-string roundtripping - Use standard DNS1123 subdomain phrasing in godocs - Fix endpoint MaxLength from 255 to 267 - Add @JoelSpeed to reviewers - Remove non-functional map key CEL validations on nodeSelector - Make replicas required (not optional) with Enum=1;3 - Use pointers with omitempty for optional scalar fields, value types with omitzero for optional struct fields per convention Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add storage immutability: `self == oldSelf` on shard Storage field and `has(oldSelf.storage) == has(self.storage)` on ManagedEtcdShardSpec to prevent adding/removing storage after creation, matching existing non-sharded etcd precedent - Add shard swap prevention: `oldSelf.all(old, self.exists(cur, cur.name == old.name))` on both managed and unmanaged Shards lists to close a gap where the size check alone couldn't prevent replacing one shard entry with another (transition rules don't fire on uncorrelated map-type list entries) - Update validation explanation to document why both size check and name preservation rules are needed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Ajpantuso

Just leaving some feedback based on a previous discussion around the appetite for this enhancement for Managed OpenShift:

The primary proposed use case makes sense for ROSA in that we would benefit from moving high churn resources to an in-memory shard
It would also make sense to exclude this in-memory shard from our disaster recovery process and set the expectation that ephemeral resources are in fact ephemeral
The primary challenge for us is transitioning existing deployments from a single etcd instance to a sharded model
- Without a means of transitioning, adoption would create a bifurcated fleet that complicates testing, adds operational overhead, confuses customers with divergent behavior based on cluster creation date rather than explicit configuration, etc...
- It could be made an explicit option, but the benefits would be difficult to describe in terms of added value. Realistically the sharded instances will just maintain optimal performance under certain usage patterns that degrade with a single instance.
- At a minimum a transition process would incur a read only period for customers which is too much of a degradation of service to consider regular maintenance.

Overall it seems like a good option to have, but would not be something for immediate adoption in ROSA. None of this feedback however should block/delay this enhancement however.

Storage type changes require data migration (PV to EmptyDir loses persistent data, EmptyDir to PV requires new volume provisioning), so the implementation correctly enforces immutability via CEL. Update the enhancement to match the implementation.

Server and peer TLS certs are per-shard, but the client TLS cert (etcd-client-tls) is shared across all shards because KAS uses a single --etcd-certfile/--etcd-keyfile for all etcd connections.

- Single-replica shards skip PDB (replicas is immutable, no lifecycle concern about needing to add it later) - TLS secrets are ~2 per shard (server + peer); client-tls is shared across all shards since KAS uses a single --etcd-certfile/keyfile - Update example calculation to match

Shard components suppress per-shard defrag RBAC manifests via predicates that return false, reusing the default etcd component's defrag ServiceAccount instead. The defrag sidecar is injected into each shard's StatefulSet with a per-shard leader election ID.

openshift-ci · 2026-07-08T12:45:50Z

@jhjaggars: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

enxebre · 2026-07-08T13:49:40Z

+  in-cluster service URLs
+- TLS credentials are generated by the PKI controller
+- The KAS deployment adapt function extends the `wait-for-etcd` init container to
+  check DNS resolution for all shard client services. The existing init container


let's make sure we don't have different headless services pointing to the same pod so we don't reintroduce openshift/hypershift#8479

enxebre · 2026-07-08T13:49:45Z

+    Replicas int32 `json:"replicas"`
+
+    // scheduling configures per-shard pod placement constraints. These
+    // constraints are merged with the framework's control plane node


I assume this one takes precedent

enxebre · 2026-07-08T13:50:18Z

thanks!
/approve

For @JoelSpeed and @csrwng to merge

openshift-ci · 2026-07-08T13:50:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~enhancements/hypershift/OWNERS~~ [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

JoelSpeed · 2026-07-08T13:50:59Z

/lgtm

/hold until I hear from Cesar

JoelSpeed · 2026-07-08T13:57:58Z

/hold cancel

openshift-ci Bot requested review from csrwng and sjenning April 22, 2026 16:11

jhjaggars force-pushed the hypershift-etcd-sharding branch 3 times, most recently from fdf69e3 to d188bb0 Compare April 23, 2026 12:36

Add enhancement: etcd sharding by resource kind for HyperShift

313f0ce

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

jhjaggars force-pushed the hypershift-etcd-sharding branch from d188bb0 to 313f0ce Compare April 23, 2026 12:48

enxebre reviewed Apr 27, 2026

View reviewed changes