From af1d3db9586ca7cdb30026ba70669083bda11cfb Mon Sep 17 00:00:00 2001 From: Jared Hayes Date: Wed, 6 Nov 2024 11:51:32 -0800 Subject: [PATCH 1/6] init --- .../ibm/service-endpoint-dynamic-override.md | 128 ++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md diff --git a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md new file mode 100644 index 0000000000..baad97787b --- /dev/null +++ b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md @@ -0,0 +1,128 @@ +--- +title: service-endpoint-dynamic-override +authors: + - jared-hayes-dev + - cjschaef + - jeffnowicki +reviewers: + - TBD +approvers: + - TBD +api-approvers: + - TBD +creation-date: 2024-11-05 +last-updated: 2024-11-05 +tracking-link: + - https://issues.redhat.com/browse/OCPCLOUD-2694 +see-also: +replaces: +superseded-by: +--- + +## Summary + +IBM Cloud wishes to support overridng service endpoints for components post cluster creation. Currently you may define overrides [prior to creating the cluster](https://github.com/openshift/installer/blob/c0938914effb0f416d01f250ea021de0cea0d690/pkg/asset/manifests/ibmcloud/cloudproviderconfig.go#L80), but the process for updating the endpoints after the creation of a cluster is not officially supported. The desire is to allow a user to configure the infrastructure object spec for IBM Cloud to specify with a list of services and endpoints to override which will be dynamically updated and reflected in all dependent components. + +## Motivation + +Management of clusters neccesitates that users be able to update endpoints should requirements/upstream services change and IBM wishes to fully support this with an official path. + +### User Stories + +* As an Openshift cluster administrator, I want to update the service endpoints for my cluster so that I can utilize the new private IAM endpoint. + +### Goals + +* Provide an official path for IBM deployed clusters to update the infrastructure object with service endpoint override(s) that will propagate to all dependent components without further user intervention. + +### Non-Goals + + + +## Proposal + +To realize this enhancement: + + +* Expand API definition to support defining services + endpoints within cloud provider spec for IBM +* Modify CCCMO so that changes are reconciled from infrastructure spec to status for IBM cloud provider and cloud config +* Update components CSI driver, ingress operator, MAPI, to pick up these changes and utilize new endpoints once they are set. + +### Workflow Description + +**cluster administrator** is a human user responsible managing an existing openshift custer deployed on IBM infrastructure. + +1. The cluster administrator wishes to use private IBM Cloud endpoints +2. The cluster administrator identifies the services that they wish to update (ie IAM and resource controller) and identifies the endpoints for these services +3. The cluster administrator updates the infrastructure object to contain a list of overrides where each element is the name of the service and the endpoint to use for that service. +4. After a delay the cluster administrator observes this change in all dependent components. + +**cccmo** is an operator responsible for watching updates to the infrastructure object and perforning updates once any value(s) are set. + +1. The cccmo reconciliation loop observes that the IBMCloudPlatform spec within the infrastructure object has been set, +2. The cccmo validates the endpoints and then updates the IBMCloudPlatformStatus and cloud config. + + +### API Extensions + +### Topology Considerations + +#### Hypershift / Hosted Control Planes + +#### Standalone Clusters + +#### Single-node Deployments or MicroShift + +### Implementation Details/Notes/Constraints + +What are some important details that didn't come across above in the +**Proposal**? Go in to as much detail as necessary here. This might be +a good place to talk about core concepts and how they relate. While it is useful +to go into the details of the code changes required, it is not necessary to show +how the code will be rewritten in the enhancement. + + +* API will be updated such that [IBMCloudPlatformSpec](https://github.com/openshift/api/blob/4c27e61e5554ea8506947d019770e5a04c3c4a36/config/v1/types_infrastructure.go#L1522) will have a field for `IBMCloudServiceEndpoints` similar to the existing field in [IBMCloudPlatformSpec](https://github.com/openshift/api/blob/4c27e61e5554ea8506947d019770e5a04c3c4a36/config/v1/types_infrastructure.go#L1549) +* CCCMO will be updated so that config sync controller via the IBM `CloudConfigTransformer` reads in endpoint settings within the spec of the infrastructure object and updates the corresponding infrastructure status and cloud config to reflect those set values. + +### Risks and Mitigations + +Users may cause service interruptions for their cluster should they define invalid overrides. This is mitigated by performing validation on the endpoint as there requirements are understood at time of implementation for IBM Cloud. + +### Drawbacks + +This change requires the cccmo manage and update the infrastructure object which is a new behavior for this operator. + +## Test Plan + +**Note:** *Section not required until targeted at a release.* + +Consider the following in developing a test plan for this enhancement: +- Will there be e2e and integration tests, in addition to unit tests? +- How will it be tested in isolation vs with other components? +- What additional testing is necessary to support managed OpenShift service-based offerings? + +No need to outline all of the test cases, just the general strategy. Anything +that would count as tricky in the implementation and anything particularly +challenging to test should be called out. + +All code is expected to have adequate tests (eventually with coverage +expectations). + +## Graduation Criteria + +### Dev Preview -> Tech Preview + +### Tech Preview -> GA + +### Removing a deprecated feature + +## Upgrade / Downgrade Strategy + +## Version Skew Strategy + +## Operational Aspects of API Extensions + +## Support Procedures + +## Alternatives \ No newline at end of file From d4882c245fd4b7973be770a0899abc683b267d0e Mon Sep 17 00:00:00 2001 From: Jared Hayes Date: Wed, 6 Nov 2024 11:55:08 -0800 Subject: [PATCH 2/6] remove instruction --- .../ibm/service-endpoint-dynamic-override.md | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md index baad97787b..f65f576df6 100644 --- a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md +++ b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md @@ -95,20 +95,6 @@ This change requires the cccmo manage and update the infrastructure object which ## Test Plan -**Note:** *Section not required until targeted at a release.* - -Consider the following in developing a test plan for this enhancement: -- Will there be e2e and integration tests, in addition to unit tests? -- How will it be tested in isolation vs with other components? -- What additional testing is necessary to support managed OpenShift service-based offerings? - -No need to outline all of the test cases, just the general strategy. Anything -that would count as tricky in the implementation and anything particularly -challenging to test should be called out. - -All code is expected to have adequate tests (eventually with coverage -expectations). - ## Graduation Criteria ### Dev Preview -> Tech Preview From 4d68f06fa411db4c970ec4614d2532953d9df1d7 Mon Sep 17 00:00:00 2001 From: Jared Hayes Date: Mon, 11 Nov 2024 13:26:04 -0800 Subject: [PATCH 3/6] update --- .../ibm/service-endpoint-dynamic-override.md | 67 ++++++------------- 1 file changed, 20 insertions(+), 47 deletions(-) diff --git a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md index f65f576df6..c2dbbc3bf1 100644 --- a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md +++ b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md @@ -21,40 +21,37 @@ superseded-by: ## Summary -IBM Cloud wishes to support overridng service endpoints for components post cluster creation. Currently you may define overrides [prior to creating the cluster](https://github.com/openshift/installer/blob/c0938914effb0f416d01f250ea021de0cea0d690/pkg/asset/manifests/ibmcloud/cloudproviderconfig.go#L80), but the process for updating the endpoints after the creation of a cluster is not officially supported. The desire is to allow a user to configure the infrastructure object spec for IBM Cloud to specify with a list of services and endpoints to override which will be dynamically updated and reflected in all dependent components. +In controlled deployments in restricted network environments, support for configuring service endpoints is required. With IBM Cloud, [support](https://docs.openshift.com/container-platform/4.17/installing/installing_ibm_cloud/installing-ibm-cloud-restricted.html#access-to-ibm-service-endpoints_installing-ibm-cloud-restricted) has already been provided to specify desired service endpoints at install time. It is also desirable to be able to change the service endpoint configuration, post install. This enhancement will extend existing support and allow post install changes to be made to the service endpoint configuration. ## Motivation -Management of clusters neccesitates that users be able to update endpoints should requirements/upstream services change and IBM wishes to fully support this with an official path. +IBM Cloud requires this enhancement for their control plane replatforming efforts. OpenShift IPI for IBM Cloud will be used to deploy a cluster with critical responsibility in our managed control plane. During the genesis phase of region bringup, existing service endpoints will be used while new regional service endpoints are brought up. Once the new regional service endpoints are available, the aforementioned cluster's service endpoint configuration will need to be updated. ### User Stories -* As an Openshift cluster administrator, I want to update the service endpoints for my cluster so that I can utilize the new private IAM endpoint. +* As an OpenShift cluster administrator, I want to update my cluster's current service endpoint configuration to point to new service endpoints. ### Goals * Provide an official path for IBM deployed clusters to update the infrastructure object with service endpoint override(s) that will propagate to all dependent components without further user intervention. -### Non-Goals - ## Proposal To realize this enhancement: - * Expand API definition to support defining services + endpoints within cloud provider spec for IBM * Modify CCCMO so that changes are reconciled from infrastructure spec to status for IBM cloud provider and cloud config -* Update components CSI driver, ingress operator, MAPI, to pick up these changes and utilize new endpoints once they are set. +* Evaluate components with service endpoint support (i.e. Storage Operator (CSI), Ingress Operator, MAPI) and ensure they react/respond to changes and utilize new service endpoint specification. In cases, where they do not, update them accordingly. ### Workflow Description **cluster administrator** is a human user responsible managing an existing openshift custer deployed on IBM infrastructure. -1. The cluster administrator wishes to use private IBM Cloud endpoints -2. The cluster administrator identifies the services that they wish to update (ie IAM and resource controller) and identifies the endpoints for these services -3. The cluster administrator updates the infrastructure object to contain a list of overrides where each element is the name of the service and the endpoint to use for that service. +1. The cluster administrator wishes to use private IBM Cloud endpoints. +2. The cluster administrator identifies the services that they wish to update (ie IAM and resource controller) and identifies the endpoints for these services. +3. The cluster administrator updates the infrastructure object to contain a list of overrides where each element is the name of the service and the endpoint to use for that service. `oc edit infrastructure -n default cluster` 4. After a delay the cluster administrator observes this change in all dependent components. **cccmo** is an operator responsible for watching updates to the infrastructure object and perforning updates once any value(s) are set. @@ -65,50 +62,26 @@ To realize this enhancement: ### API Extensions -### Topology Considerations - -#### Hypershift / Hosted Control Planes +* Extend `IBMCloudPlatformSpec` to contain service endpoint field that users may define as desired overrides. -#### Standalone Clusters +``` +type IBMCloudPlatformSpec struct { + // serviceEndpoints is a list of custom endpoints which will override the default + // service endpoints of an IBM Cloud service. These endpoints are consumed by + // components within the cluster to reach the respective IBM Cloud Services. + // +listType=map + // +listMapKey=name + // +optional + ServiceEndpoints []IBMCloudServiceEndpoint `json:"serviceEndpoints,omitempty"` +} -#### Single-node Deployments or MicroShift +``` ### Implementation Details/Notes/Constraints -What are some important details that didn't come across above in the -**Proposal**? Go in to as much detail as necessary here. This might be -a good place to talk about core concepts and how they relate. While it is useful -to go into the details of the code changes required, it is not necessary to show -how the code will be rewritten in the enhancement. - - * API will be updated such that [IBMCloudPlatformSpec](https://github.com/openshift/api/blob/4c27e61e5554ea8506947d019770e5a04c3c4a36/config/v1/types_infrastructure.go#L1522) will have a field for `IBMCloudServiceEndpoints` similar to the existing field in [IBMCloudPlatformSpec](https://github.com/openshift/api/blob/4c27e61e5554ea8506947d019770e5a04c3c4a36/config/v1/types_infrastructure.go#L1549) * CCCMO will be updated so that config sync controller via the IBM `CloudConfigTransformer` reads in endpoint settings within the spec of the infrastructure object and updates the corresponding infrastructure status and cloud config to reflect those set values. ### Risks and Mitigations -Users may cause service interruptions for their cluster should they define invalid overrides. This is mitigated by performing validation on the endpoint as there requirements are understood at time of implementation for IBM Cloud. - -### Drawbacks - -This change requires the cccmo manage and update the infrastructure object which is a new behavior for this operator. - -## Test Plan - -## Graduation Criteria - -### Dev Preview -> Tech Preview - -### Tech Preview -> GA - -### Removing a deprecated feature - -## Upgrade / Downgrade Strategy - -## Version Skew Strategy - -## Operational Aspects of API Extensions - -## Support Procedures - -## Alternatives \ No newline at end of file +Users may cause service interruptions for their cluster should they define invalid overrides. This is mitigated by performing validation on the endpoint as there requirements are understood at time of implementation for IBM Cloud. \ No newline at end of file From 565d917da3d34c73820e74d14249c24319dc2071 Mon Sep 17 00:00:00 2001 From: Jared Hayes Date: Thu, 14 Nov 2024 09:53:10 -0800 Subject: [PATCH 4/6] fix lint --- .../ibm/service-endpoint-dynamic-override.md | 46 +++++++++++++++++-- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md index c2dbbc3bf1..68a0af6826 100644 --- a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md +++ b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md @@ -19,6 +19,8 @@ replaces: superseded-by: --- +# IBM Cloud Service Endpoint Override Support + ## Summary In controlled deployments in restricted network environments, support for configuring service endpoints is required. With IBM Cloud, [support](https://docs.openshift.com/container-platform/4.17/installing/installing_ibm_cloud/installing-ibm-cloud-restricted.html#access-to-ibm-service-endpoints_installing-ibm-cloud-restricted) has already been provided to specify desired service endpoints at install time. It is also desirable to be able to change the service endpoint configuration, post install. This enhancement will extend existing support and allow post install changes to be made to the service endpoint configuration. @@ -43,7 +45,7 @@ To realize this enhancement: * Expand API definition to support defining services + endpoints within cloud provider spec for IBM * Modify CCCMO so that changes are reconciled from infrastructure spec to status for IBM cloud provider and cloud config -* Evaluate components with service endpoint support (i.e. Storage Operator (CSI), Ingress Operator, MAPI) and ensure they react/respond to changes and utilize new service endpoint specification. In cases, where they do not, update them accordingly. +* Evaluate IBM components with service endpoint support and how they will react/respond to post-install service endpoint changes. ### Workflow Description @@ -74,9 +76,22 @@ type IBMCloudPlatformSpec struct { // +optional ServiceEndpoints []IBMCloudServiceEndpoint `json:"serviceEndpoints,omitempty"` } - ``` +### Topology Considerations + +#### Hypershift / Hosted Control Planes + +n/a + +#### Standalone Clusters + +n/a + +#### Single-node Deployments or MicroShift + +n/a + ### Implementation Details/Notes/Constraints * API will be updated such that [IBMCloudPlatformSpec](https://github.com/openshift/api/blob/4c27e61e5554ea8506947d019770e5a04c3c4a36/config/v1/types_infrastructure.go#L1522) will have a field for `IBMCloudServiceEndpoints` similar to the existing field in [IBMCloudPlatformSpec](https://github.com/openshift/api/blob/4c27e61e5554ea8506947d019770e5a04c3c4a36/config/v1/types_infrastructure.go#L1549) @@ -84,4 +99,29 @@ type IBMCloudPlatformSpec struct { ### Risks and Mitigations -Users may cause service interruptions for their cluster should they define invalid overrides. This is mitigated by performing validation on the endpoint as there requirements are understood at time of implementation for IBM Cloud. \ No newline at end of file +Users may cause service interruptions for their cluster should they define invalid overrides. This is mitigated by performing validation on the endpoint as there requirements are understood at time of implementation for IBM Cloud. + + +### Drawbacks + +n/a + +## Test Plan + +## Graduation Criteria + +### Dev Preview -> Tech Preview + +### Tech Preview -> GA + +### Removing a deprecated feature + +## Upgrade / Downgrade Strategy + +## Version Skew Strategy + +## Operational Aspects of API Extensions + +## Support Procedures + +## Alternatives \ No newline at end of file From 737a7ae1501ee4debbe2274120513504104f4a6f Mon Sep 17 00:00:00 2001 From: Jared Hayes Date: Thu, 14 Nov 2024 10:07:30 -0800 Subject: [PATCH 5/6] wording --- .../cloud-integration/ibm/service-endpoint-dynamic-override.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md index 68a0af6826..6fb6052e67 100644 --- a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md +++ b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md @@ -54,7 +54,7 @@ To realize this enhancement: 1. The cluster administrator wishes to use private IBM Cloud endpoints. 2. The cluster administrator identifies the services that they wish to update (ie IAM and resource controller) and identifies the endpoints for these services. 3. The cluster administrator updates the infrastructure object to contain a list of overrides where each element is the name of the service and the endpoint to use for that service. `oc edit infrastructure -n default cluster` -4. After a delay the cluster administrator observes this change in all dependent components. +4. Once the service endpoint override update has been processed/reconciled, components can act on the change (if applicable) and use in future operations (note: may need to be restarted to pick up the change). **cccmo** is an operator responsible for watching updates to the infrastructure object and perforning updates once any value(s) are set. From 9281a121bb7cf1f9bc982fb1d911f33f1c52d9f8 Mon Sep 17 00:00:00 2001 From: Jared Hayes Date: Mon, 18 Nov 2024 14:38:26 -0800 Subject: [PATCH 6/6] pr comments --- .../cloud-integration/ibm/service-endpoint-dynamic-override.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md index 6fb6052e67..44b769442a 100644 --- a/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md +++ b/enhancements/cloud-integration/ibm/service-endpoint-dynamic-override.md @@ -31,7 +31,7 @@ IBM Cloud requires this enhancement for their control plane replatforming effort ### User Stories -* As an OpenShift cluster administrator, I want to update my cluster's current service endpoint configuration to point to new service endpoints. +* As an OpenShift cluster administrator, I want to update my cluster's current service endpoint configuration to point to new service endpoints so that I can comply with administrative requirements to directs component traffic through regional or private endpoints. ### Goals @@ -45,7 +45,6 @@ To realize this enhancement: * Expand API definition to support defining services + endpoints within cloud provider spec for IBM * Modify CCCMO so that changes are reconciled from infrastructure spec to status for IBM cloud provider and cloud config -* Evaluate IBM components with service endpoint support and how they will react/respond to post-install service endpoint changes. ### Workflow Description