From bb5cd7b73d79277d19e370e90fcc83ea5bdec46d Mon Sep 17 00:00:00 2001 From: Bharath B Date: Mon, 18 Jul 2022 10:37:00 +0530 Subject: [PATCH 01/27] Apply user defined tags to all Azure resources created by OpenShift --- azure_user_defined_tags.md | 409 +++++++++++++++++++++++++++++++++++++ 1 file changed, 409 insertions(+) create mode 100644 azure_user_defined_tags.md diff --git a/azure_user_defined_tags.md b/azure_user_defined_tags.md new file mode 100644 index 0000000000..4a455fd106 --- /dev/null +++ b/azure_user_defined_tags.md @@ -0,0 +1,409 @@ +--- +title: apply-user-defined-tags-to-all-azure-resources-created-by-openShift +authors: + - "@bhb" +reviewers: + - +approvers: + - +api-approvers: + - +creation-date: 2022-07-12 +last-updated: 2022-07-12 +tracking-link: + - https://issues.redhat.com/browse/RFE-2017 + - https://issues.redhat.com/browse/CFE-492 + - https://issues.redhat.com/browse/CFEPLAN-59 +see-also: + - +replaces: + - +superseded-by: + - +--- + +# Apply user defined tags to all Azure resources created by OpenShift + +## Summary + +This enhancement describes the proposal to allow an administrator of Openshift to +have the ability to apply user defined tags to those resources created by Openshift +in Azure. + +## Motivation + +Motivations include but are not limited to: + +- Allow admin, compliance, and security teams to keep track of assets and objects + created by OpenShift, both at install time and during continuous operation (Day 2) + +### User Stories + +- As an openshift administrator, I want to have tags added to all resources created + in Azure by Openshift on Day 1. +- As an openshift administrator, I want to restrict access granted to Openshift specific account. +- As an openshift administrator, I should be able to create new tags or modify existing tags. + +### Goals + +- The administrator or service (in the case of Managed OpenShift) installing OpenShift + can pass an arbitrary list of user-defined tags to the OpenShift installer, and + the installer and all the bootstrapped components will apply these tags to all + resources created in Azure, for the life of the cluster, and where supported by Azure. +- Tags must be applied at creation time, in an atomic operation. It isn't acceptable + to create an object and, after some period of time, apply the tags post-creation. + +### Non-Goals + +- To reduce initial scope, support for deleting the tags is not supported. + +## Proposal + +A tag of the form `kubernetes.io/cluster/:owned` will be added to every +resource created by Openshift to enable administrator to differentiate the resources +created for Openshift cluster. An administrator is not allowed to add or modify the tag +having the prefix `kubernetes.io` in the name. + +New `userTags` field will be added to `platform.azure` of install-config for the user +to define the tags to be added to the resources created by installer and in-cluster operators. + +If `platform.azure.userTags` of install-config has any tag defined same will be added +to all the azure resources created by Openshift except, when the tag validation fails +due to any of below conditions + - A tag name can have a maximum of 128 characters + (Note: Tag name has a limit of 512 characters for all resources except for + storage accounts, which has a limit of 128 characters and hence tag name + length is restricted to 128 characters for every resource required by Openshift) + - A tag value has a limit of 256 characters. + - A tag name cannot contain `<, >, %, &, \, ?, /, #, :, whitespace` characters and + should not start with a number. + (Note: DNS zones, Traffic, Front Door resources does not support tag with spaces, + special/unicode characters or starting with number, hence these are added as + constraints to every other azure resource required by Openshift as well.) + - A resource, resource-group or subscription can have a maximum of 10 tags. + (Note: Azure supports a maximum of 50 tags except for Automation, + Content Delivery Network, DNS resources which can have a maximum of 15 tags, hence + restricting the number of tags to 10 for all resources created by Openshift, with 5 + spared for administrator's use) + +Add a new field `resourceTags` to `.spec.platformSpec.azure` of the +`infrastructure.config.openshift.io` type. Tags included in the `resourceTags` field +will be applied to new resources created for the cluster. + +Add a new field `resourceTags` to `.status.platformStatus.azure` of the +`infrastructure.config.openshift.io` type. `resourceTags` will have the information +on the resources for which tag operation failed, updated by the respective in-cluster operator. + +All operators that create Azure resources (Cluster Infrastructure ,Storage ,Node ,NetworkEdge , +Internal Registry ,CCO) will apply these tags to all Azure resources they create. + +`resourceTags` that are specified in the infrastructure resource will merge with tags +specified in an Azure resource. In the case where `resourceTags` specified in the +infrastructure resource and there is a tag with the same name specified in an Azure +resource, the value from the infrastructure resource will take precedence and be updated. + +The userTags field is intended to be set at install time and is considered immutable. +Components that respect this field must only ever add tags that they retrieve from this +field to cloud resources, they must never remove tags from the existing underlying cloud +resource even if the tags are removed from this field(despite it being immutable). + +If the userTags field is changed post-install, there is no guarantee about how an +in-cluster operator will respond to the change. Some operators may reconcile the +change and change tags on the Azure resource. Some operators may ignore the change. +However, if tags are removed from userTags, the tag will not be removed from the +Azure resource. + +### Workflow Description + +- An Openshift administrator requests to add required tags to all Azure resources + created by Openshift by adding it in `.platform.azure.userTags` +- openshift installer validates the tags defined in `.platform.azure.userTags` and + adds these tags to all resources created during installation and also updates + `.spec.platformSpec.azure.resourceTags` of the `infrastructure.config.openshift.io` +- In cluster operators refers `.spec.platformSpec.azure.resourceTags` of the + `infrastructure.config.openshift.io` to add tags to resources created later. +- An Openshift administrator can modify existing tags or add new tags by updating + `.spec.platformSpec.azure.resourceTags` field in the `infrastructure.config.openshift.io`. + +#### Variation [optional] + +### API Extensions +Enhancement requires below modifications to the mentioned CRDs +- Add `userTags` field to `platform.azure` of the `installconfigs.install.openshift.io` +```yaml + apiVersion: apiextensions.k8s.io/v1 + kind: CustomResourceDefinition + metadata: + name: installconfigs.install.openshift.io + spec: + versions: + - name: v1 + schema: + openAPIV3Schema: + description: InstallConfig is the configuration for an OpenShift install. + properties: + platform: + description: Platform is configuration for machine pool specific + to the platform. + properties: + azure: + description: Azure is the configuration used when installing + on Azure. + properties: + userTags: + additionalProperties: + type: string + description: UserTags additional keys and values that the installer + will add as tags to all resources that it creates. Resources + created by the cluster itself may not include these tags. + type: object +``` + +- Add `resourceTags` field to `spec.platformSpec.azure` and `platformStatus.status.azure` + of the `infrastructure.config.openshift.io` +```yaml + apiVersion: apiextensions.k8s.io/v1 + kind: CustomResourceDefinition + metadata: + name: infrastructures.config.openshift.io + spec: + versions: + - name: v1 + schema: + openAPIV3Schema: + description: "Infrastructure holds cluster-wide information about Infrastructure. The canonical name is `cluster` \n Compatibility level 1: Stable within a major release for a minimum of 12 months or 3 minor releases (whichever is longer)." + properties: + spec: + description: spec holds user settable values for configuration + properties: + platformSpec: + description: platformSpec holds desired information specific to the underlying infrastructure provider. + properties: + azure: + description: Azure contains settings specific to the Azure infrastructure provider. + properties: + resourceTags: + description: resourceTags is a list of additional tags to apply to Azure resources created for the cluster. See https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few. OpenShift reserves 10 tags for its use, leaving 40 tags available for the user. + type: array + maxItems: 10 + items: + description: AzureResourceTag is a tag to apply to Azure resources created for the cluster. + type: object + required: + - key + - value + properties: + key: + description: key is the key of the tag + type: string + maxLength: 128 + minLength: 1 + pattern: ^[a-zA-Z][0-9A-Za-z_.=+-@]+$ + value: + description: value is the value of the tag. + type: string + maxLength: 256 + minLength: 1 + pattern: ^[0-9A-Za-z_.=+-@]+$ + status: + description: status holds observed values from the cluster. They may not be overridden. + properties: + platformStatus: + description: platformStatus holds status information specific to the underlying infrastructure provider. + properties: + azure: + description: Azure contains settings specific to the Azure infrastructure provider. + properties: + resourceTags: + description: resourceTags is a list of Azure resources for which tag update got failed. It contains the Azure resource name and the error encountered. + type: array + items: + description: Azure resource is the name of the resource created for the cluster. + type: object + required: + - name + - error + properties: + name: + description: name is the name of the Azure resource. + type: string + error: + description: error is the reason for the tag update failure. + type: string +``` + +### Implementation Details/Notes/Constraints [optional] + +What are the caveats to the implementation? What are some important details that +didn't come across above. Go in to as much detail as necessary here. This might +be a good place to talk about core concepts and how they relate. + +### Risks and Mitigations + +### Drawbacks +- User-defined tags cannot be updated on an Azure resource which is not managed by an + operator. In this proposal, the changes proposed and developed will be part of + openshift-* namespace. External operators are not in scope. + User-defined tags can be updated on the following Azure resources. + 1. Virtual machine resources created for master and worker nodes. + 2. Storage account + 3. NetworkEdge + 4. Internal Registry + 5. CCO + +- OpenShift is bound to have the common limitation for all Azure resources created + by it and constrains other resources with the least matching limit as below + 1. Tag names cannot have `microsoft`, `azure`, `windows` prefixes which are + reserved for Azure use. + 2. An Azure storage account has a limit of 128 characters for the tag name. + 3. An Azure DNS zone or Traffic or Front Door resource tag name cannot have spaces, + special/unicode characters or start with a number. + 4. An Azure Automation or Content Delivery Network or DNS resource can have a + maximum of 15 tags. + +- Administrator will have to manually perform tags pertaining actions for + - removing the undesired tags from the required resources. + - update tags of the resources which are not managed by an operator. + - update tags of the resources for which update logic is not supported by an operator. + +## Design Details +The `resourceTags` field in `spec.platformSpec.azure` will be populated by the +installer using the entries from `userTags` field. + +`infrastructure.config.openshift.io` fields `spec.platformSpec.azure` is a mutable +and `status.platformStatus.azure` is immutable. + +All operators that create Azure resources (Cluster Infrastructure ,Storage ,Node , +NetworkEdge ,Internal Registry ,CCO) will apply these tags to all Azure resources they create. + +Operators should create an event to indicate the status of tag add/modify request and +as well update the `status.platformSpec.azure` field of `infrastructure.config.openshift.io` +with required details in case of failure. + +#### Create tags scenarios +In the cases where a tag is specified in the `infrastructure.config.openshift.io` resource and +1) A tag with the same key is present in an Azure resource, the value will be replaced with + `infrastructure.config.openshift.io` resource value + + For example, + Azure resource as a tag `key_infra = value_azure` + New tag request = `spec.platformSpec.azure.resourceTags` has `key_infra = value_infra` + Action = Azure resource tag is updated to `key_infra = value_infra` + Event action = An event is generated to notify user about the request + status (success/failure) to update tags for the Azure resource. + +2) A tag with same key is not present and is a new tag to be added for an Aure + resource, but maximum tag limit is reached or any other error encountered + should be notified to user. + + For example, + New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_infra` + Action = A new tag to be added for an Azure resource. + Final tag set to Azure resource = `key_infra = value_infra` + Event action = An event is generated to notify user about the request + status (success/failure) to create tags for the Azure resource. + +#### Update tags scenarios +Users can update the user-defined tags by editing `.spec.platformSpec.azure` of the +`infrastructure.config.openshift.io` resource. On update of `resourceTags`, +Azure resource is not created or restarted. + +In the case where a tag is updated in the `infrastructure.config.openshift.io` resource and +1) A tag with the same key and value is present in an Azure resource, no update is made. + + For example, + Existing tag for Azure resource = `key_infra = value_update` + New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_update` + Action = There is no update for Azure resource. + Final tag set to Azure resource = `key_infra = value_update` + +2) A tag with the same key but different value is present in an Azure resource, the Azure + resource will be updated with the new value. + + For example, + Existing tag for Azure resource = `key_infra = value` + New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_new` + Action = Existing tag for Azure resource is updated to reflect new value. + Final tag set to Azure resource = `key_infra = value_new` + Event action = An event is generated to notify user about the request + status (success/failure) to update tags for the Azure resource. + +#### Caveats +1) User updates Azure resource's tag using an external interface which is present in + `.spec.platformSpec.azure.resourceTags`, tag will be reconciled by the owning operator + with the value from the `.spec.platformSpec.azure.resourceTags`. Reconciliation does not + happen set immediately for Azure resource by the owning operator. There is eventual + consistency maintained by the owning operator. The time taken to reconcile the modified + tag on Azure resource to desired value vary across owning operators. + + For example, + Edited existing tag using external tool for Azure resource = `key_infra = value_tool` + Previous tag request = `.spec.platformSpec.azure.resourceTags` or resource has `key_infra = value` + Action = Update existing tag with value from `.spec.platformSpec.azure.resourceTags`. + Final tag set to Azure resource = `key_infra1 = value` + Event action = An event is generated to notify user about the request + status (success/failure) to update tags for the Azure resource. + +2) User removes the user-defined tag from `.spec.platformSpec.azure.resourceTags` or Azure resource. + The user-defined tag which is removed from spec will not be reconciled or managed by operators. + User can update tag using an external interface. The user-defined tag will not be modified + by the operator. + + For example, + Existing tag for Azure resource = `key_infra = value` + New tag request = `.spec.platformSpec.azure.resourceTags` has no user-defined tag with key `key_infra` + Action = No change in tags for the Azure resource. + Final tag set to Azure resource = `key_infra1 = value` + +3) Updating tags of individual resources is not supported and any tag present in + `.spec.platformSpec.azure.resourceTags` of `infrastructure.config.openshift.io/v1` resource + will result in updating tags of all Openshift managed Azure resources. + +### Open Questions + +### Test Plan + +### Graduation Criteria + +#### Dev Preview -> Tech Preview + +#### Tech Preview -> GA + +#### Removing a deprecated feature + +### Upgrade / Downgrade Strategy + +On upgrade: +- Cluster operators that update the tags of Azure resources created for cluster + should refer the new fields and take action. + +On downgrade: +- The status/spec field may remain populated, components may or may not continue + to tag newly created resources with the additional tags depending on whether or + not a given component still has logic to respect the status tags, after the downgrade. + +### Version Skew Strategy + +### Operational Aspects of API Extensions + +#### Failure Modes + +#### Support Procedures + +## Implementation History + +## Alternatives +Alternate or extended proposal is to have a dedicated controller for managing +the cloud infrastructure as desired by the user by supporting but not limited +to adding or modifying tags of the cloud resources created by Openshift. + +### Motivation +Motivations for having a dedicated cloud-infra-controller include but are not limited to +- A single controller which adds/modifies tags of the all resources created by + Openshift and updates the status of the operation in `infrastructure.config.openshift.io` + instead of multiple controllers doing so. +- Having single controller will solve the issue of tags not getting added to those resources + created by installer and not being managed by any operator during tag update or cluster + upgrade scenario(from feature unsupported version). + +## Infrastructure Needed [optional] \ No newline at end of file From 7ef81e76aaabadaac63397bfe64c827efad2f574 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Mon, 18 Jul 2022 13:25:03 +0530 Subject: [PATCH 02/27] format content for better reading --- azure_user_defined_tags.md | 114 ++++++++++++++++++++++--------------- 1 file changed, 67 insertions(+), 47 deletions(-) diff --git a/azure_user_defined_tags.md b/azure_user_defined_tags.md index 4a455fd106..c3e4394faa 100644 --- a/azure_user_defined_tags.md +++ b/azure_user_defined_tags.md @@ -69,22 +69,22 @@ to define the tags to be added to the resources created by installer and in-clus If `platform.azure.userTags` of install-config has any tag defined same will be added to all the azure resources created by Openshift except, when the tag validation fails -due to any of below conditions - - A tag name can have a maximum of 128 characters - (Note: Tag name has a limit of 512 characters for all resources except for - storage accounts, which has a limit of 128 characters and hence tag name - length is restricted to 128 characters for every resource required by Openshift) - - A tag value has a limit of 256 characters. - - A tag name cannot contain `<, >, %, &, \, ?, /, #, :, whitespace` characters and - should not start with a number. - (Note: DNS zones, Traffic, Front Door resources does not support tag with spaces, - special/unicode characters or starting with number, hence these are added as - constraints to every other azure resource required by Openshift as well.) - - A resource, resource-group or subscription can have a maximum of 10 tags. - (Note: Azure supports a maximum of 50 tags except for Automation, - Content Delivery Network, DNS resources which can have a maximum of 15 tags, hence - restricting the number of tags to 10 for all resources created by Openshift, with 5 - spared for administrator's use) +due to any of below conditions
+1. A tag name can have a maximum of 128 characters
+ (Note: Tag name has a limit of 512 characters for all resources except for + storage accounts, which has a limit of 128 characters and hence tag name + length is restricted to 128 characters for every resource required by Openshift)
+2. A tag value has a limit of 256 characters.
+3. A tag name cannot contain `<, >, %, &, \, ?, /, #, :, whitespace` characters and + should not start with a number.
+ (Note: DNS zones, Traffic, Front Door resources does not support tag with spaces, + special/unicode characters or starting with number, hence these are added as + constraints to every other azure resource required by Openshift as well.)
+4. A resource, resource-group or subscription can have a maximum of 10 tags.
+ (Note: Azure supports a maximum of 50 tags except for Automation, + Content Delivery Network, DNS resources which can have a maximum of 15 tags, hence + restricting the number of tags to 10 for all resources created by Openshift, with 5 + spared for administrator's use)
Add a new field `resourceTags` to `.spec.platformSpec.azure` of the `infrastructure.config.openshift.io` type. Tags included in the `resourceTags` field @@ -262,9 +262,9 @@ be a good place to talk about core concepts and how they relate. maximum of 15 tags. - Administrator will have to manually perform tags pertaining actions for - - removing the undesired tags from the required resources. - - update tags of the resources which are not managed by an operator. - - update tags of the resources for which update logic is not supported by an operator. + 1. removing the undesired tags from the required resources. + 2. update tags of the resources which are not managed by an operator. + 3. update tags of the resources for which update logic is not supported by an operator. ## Design Details The `resourceTags` field in `spec.platformSpec.azure` will be populated by the @@ -285,10 +285,10 @@ In the cases where a tag is specified in the `infrastructure.config.openshift.io 1) A tag with the same key is present in an Azure resource, the value will be replaced with `infrastructure.config.openshift.io` resource value - For example, - Azure resource as a tag `key_infra = value_azure` - New tag request = `spec.platformSpec.azure.resourceTags` has `key_infra = value_infra` - Action = Azure resource tag is updated to `key_infra = value_infra` + For example,
+ Azure resource as a tag `key_infra = value_azure`
+ New tag request = `spec.platformSpec.azure.resourceTags` has `key_infra = value_infra`
+ Action = Azure resource tag is updated to `key_infra = value_infra`
Event action = An event is generated to notify user about the request status (success/failure) to update tags for the Azure resource. @@ -296,10 +296,10 @@ In the cases where a tag is specified in the `infrastructure.config.openshift.io resource, but maximum tag limit is reached or any other error encountered should be notified to user. - For example, - New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_infra` - Action = A new tag to be added for an Azure resource. - Final tag set to Azure resource = `key_infra = value_infra` + For example,
+ New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_infra`
+ Action = A new tag to be added for an Azure resource.
+ Final tag set to Azure resource = `key_infra = value_infra`
Event action = An event is generated to notify user about the request status (success/failure) to create tags for the Azure resource. @@ -311,20 +311,20 @@ Azure resource is not created or restarted. In the case where a tag is updated in the `infrastructure.config.openshift.io` resource and 1) A tag with the same key and value is present in an Azure resource, no update is made. - For example, - Existing tag for Azure resource = `key_infra = value_update` - New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_update` - Action = There is no update for Azure resource. + For example,
+ Existing tag for Azure resource = `key_infra = value_update`
+ New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_update`
+ Action = There is no update for Azure resource.
Final tag set to Azure resource = `key_infra = value_update` 2) A tag with the same key but different value is present in an Azure resource, the Azure resource will be updated with the new value. - For example, - Existing tag for Azure resource = `key_infra = value` - New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_new` - Action = Existing tag for Azure resource is updated to reflect new value. - Final tag set to Azure resource = `key_infra = value_new` + For example,
+ Existing tag for Azure resource = `key_infra = value`
+ New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_new`
+ Action = Existing tag for Azure resource is updated to reflect new value.
+ Final tag set to Azure resource = `key_infra = value_new`
Event action = An event is generated to notify user about the request status (success/failure) to update tags for the Azure resource. @@ -336,11 +336,11 @@ In the case where a tag is updated in the `infrastructure.config.openshift.io` r consistency maintained by the owning operator. The time taken to reconcile the modified tag on Azure resource to desired value vary across owning operators. - For example, - Edited existing tag using external tool for Azure resource = `key_infra = value_tool` - Previous tag request = `.spec.platformSpec.azure.resourceTags` or resource has `key_infra = value` - Action = Update existing tag with value from `.spec.platformSpec.azure.resourceTags`. - Final tag set to Azure resource = `key_infra1 = value` + For example,
+ Edited existing tag using external tool for Azure resource = `key_infra = value_tool`
+ Previous tag request = `.spec.platformSpec.azure.resourceTags` or resource has `key_infra = value`
+ Action = Update existing tag with value from `.spec.platformSpec.azure.resourceTags`.
+ Final tag set to Azure resource = `key_infra1 = value`
Event action = An event is generated to notify user about the request status (success/failure) to update tags for the Azure resource. @@ -349,11 +349,11 @@ In the case where a tag is updated in the `infrastructure.config.openshift.io` r User can update tag using an external interface. The user-defined tag will not be modified by the operator. - For example, - Existing tag for Azure resource = `key_infra = value` - New tag request = `.spec.platformSpec.azure.resourceTags` has no user-defined tag with key `key_infra` - Action = No change in tags for the Azure resource. - Final tag set to Azure resource = `key_infra1 = value` + For example,
+ Existing tag for Azure resource = `key_infra = value`
+ New tag request = `.spec.platformSpec.azure.resourceTags` has no user-defined tag with key `key_infra`
+ Action = No change in tags for the Azure resource.
+ Final tag set to Azure resource = `key_infra1 = value`
3) Updating tags of individual resources is not supported and any tag present in `.spec.platformSpec.azure.resourceTags` of `infrastructure.config.openshift.io/v1` resource @@ -406,4 +406,24 @@ Motivations for having a dedicated cloud-infra-controller include but are not li created by installer and not being managed by any operator during tag update or cluster upgrade scenario(from feature unsupported version). -## Infrastructure Needed [optional] \ No newline at end of file +### Design Details +- The dedicated controller will watch for changes in the `infrastructure.config.openshift.io` + resource and acts only when there is a change to `spec.platformSpec.azure` field. +- Controller queries for all the resources which has the tag + `kubernetes.io/cluster/:owned` and updates each resource with the new + requested changes. +- In case cluster is upgraded from a release without Azure tag implementation (query for + resources with `kubernetes.io/cluster/:owned` tag yields just resource group + type), controller queries for the resources having cluster name in the name tag and + updates each matching resource with the new requested changes. +- In-cluster operators should not watch for changes to `spec.platformSpec.azure` field of + `infrastructure.config.openshift.io` and should continue with current functionality of + creating requested resources, but include additional functionality to add user defined tags + and the `kubernetes.io/cluster/:owned` default tag to created resource. + +### Caveats +- Few Azure resources such as disk, storage account, DNS zones and records name might not + match either `kubernetes.io/cluster/:owned` or cluster name in the name tag + and will result in tagging inconsistency, which will be minimal compared not to having + aformentioned dedicated controller for managing tags of Azure resources. +## Infrastructure Needed [optional] From 77da16f2c3bd306133dce6716e6adcf32e9e6785 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Mon, 18 Jul 2022 14:06:02 +0530 Subject: [PATCH 03/27] update indentation of sample yaml --- azure_user_defined_tags.md | 226 +++++++++++++++++++++---------------- 1 file changed, 128 insertions(+), 98 deletions(-) diff --git a/azure_user_defined_tags.md b/azure_user_defined_tags.md index c3e4394faa..e6032ae04b 100644 --- a/azure_user_defined_tags.md +++ b/azure_user_defined_tags.md @@ -131,105 +131,105 @@ Azure resource. Enhancement requires below modifications to the mentioned CRDs - Add `userTags` field to `platform.azure` of the `installconfigs.install.openshift.io` ```yaml - apiVersion: apiextensions.k8s.io/v1 - kind: CustomResourceDefinition - metadata: - name: installconfigs.install.openshift.io - spec: - versions: - - name: v1 - schema: - openAPIV3Schema: - description: InstallConfig is the configuration for an OpenShift install. - properties: - platform: - description: Platform is configuration for machine pool specific - to the platform. - properties: - azure: - description: Azure is the configuration used when installing - on Azure. - properties: - userTags: - additionalProperties: - type: string - description: UserTags additional keys and values that the installer - will add as tags to all resources that it creates. Resources - created by the cluster itself may not include these tags. - type: object +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: installconfigs.install.openshift.io +spec: + versions: + - name: v1 + schema: + openAPIV3Schema: + description: InstallConfig is the configuration for an OpenShift install. + properties: + platform: + description: Platform is configuration for machine pool specific + to the platform. + properties: + azure: + description: Azure is the configuration used when installing + on Azure. + properties: + userTags: + additionalProperties: + type: string + description: UserTags additional keys and values that the installer + will add as tags to all resources that it creates. Resources + created by the cluster itself may not include these tags. + type: object ``` - Add `resourceTags` field to `spec.platformSpec.azure` and `platformStatus.status.azure` of the `infrastructure.config.openshift.io` ```yaml - apiVersion: apiextensions.k8s.io/v1 - kind: CustomResourceDefinition - metadata: - name: infrastructures.config.openshift.io - spec: - versions: - - name: v1 - schema: - openAPIV3Schema: - description: "Infrastructure holds cluster-wide information about Infrastructure. The canonical name is `cluster` \n Compatibility level 1: Stable within a major release for a minimum of 12 months or 3 minor releases (whichever is longer)." - properties: - spec: - description: spec holds user settable values for configuration - properties: - platformSpec: - description: platformSpec holds desired information specific to the underlying infrastructure provider. - properties: - azure: - description: Azure contains settings specific to the Azure infrastructure provider. - properties: - resourceTags: - description: resourceTags is a list of additional tags to apply to Azure resources created for the cluster. See https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few. OpenShift reserves 10 tags for its use, leaving 40 tags available for the user. - type: array - maxItems: 10 - items: - description: AzureResourceTag is a tag to apply to Azure resources created for the cluster. - type: object - required: - - key - - value - properties: - key: - description: key is the key of the tag - type: string - maxLength: 128 - minLength: 1 - pattern: ^[a-zA-Z][0-9A-Za-z_.=+-@]+$ - value: - description: value is the value of the tag. - type: string - maxLength: 256 - minLength: 1 - pattern: ^[0-9A-Za-z_.=+-@]+$ - status: - description: status holds observed values from the cluster. They may not be overridden. - properties: - platformStatus: - description: platformStatus holds status information specific to the underlying infrastructure provider. - properties: - azure: - description: Azure contains settings specific to the Azure infrastructure provider. - properties: - resourceTags: - description: resourceTags is a list of Azure resources for which tag update got failed. It contains the Azure resource name and the error encountered. - type: array - items: - description: Azure resource is the name of the resource created for the cluster. - type: object - required: - - name - - error - properties: - name: - description: name is the name of the Azure resource. - type: string - error: - description: error is the reason for the tag update failure. - type: string +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: infrastructures.config.openshift.io +spec: + versions: + - name: v1 + schema: + openAPIV3Schema: + description: "Infrastructure holds cluster-wide information about Infrastructure. The canonical name is `cluster` \n Compatibility level 1: Stable within a major release for a minimum of 12 months or 3 minor releases (whichever is longer)." + properties: + spec: + description: spec holds user settable values for configuration + properties: + platformSpec: + description: platformSpec holds desired information specific to the underlying infrastructure provider. + properties: + azure: + description: Azure contains settings specific to the Azure infrastructure provider. + properties: + resourceTags: + description: resourceTags is a list of additional tags to apply to Azure resources created for the cluster. See https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few. OpenShift reserves 10 tags for its use, leaving 40 tags available for the user. + type: array + maxItems: 10 + items: + description: AzureResourceTag is a tag to apply to Azure resources created for the cluster. + type: object + required: + - key + - value + properties: + key: + description: key is the key of the tag + type: string + maxLength: 128 + minLength: 1 + pattern: ^[a-zA-Z][0-9A-Za-z_.=+-@]+$ + value: + description: value is the value of the tag. + type: string + maxLength: 256 + minLength: 1 + pattern: ^[0-9A-Za-z_.=+-@]+$ + status: + description: status holds observed values from the cluster. They may not be overridden. + properties: + platformStatus: + description: platformStatus holds status information specific to the underlying infrastructure provider. + properties: + azure: + description: Azure contains settings specific to the Azure infrastructure provider. + properties: + resourceTags: + description: resourceTags is a list of Azure resources for which tag update got failed. It contains the Azure resource name and the encountered error. + type: array + items: + description: Azure resource is the name of the resource created for the cluster. + type: object + required: + - name + - error + properties: + name: + description: name is the name of the Azure resource. + type: string + error: + description: error is the reason for the tag update failure. + type: string ``` ### Implementation Details/Notes/Constraints [optional] @@ -280,6 +280,35 @@ Operators should create an event to indicate the status of tag add/modify reques as well update the `status.platformSpec.azure` field of `infrastructure.config.openshift.io` with required details in case of failure. +Below list of terraform Azure APIs to create resources should be updated to add user +defined tags and as well the openshift default tag in the installer component. +`azurerm_resource_group, azurerm_image, azurerm_lb, azurerm_lb_backend_address_pool, +azurerm_lb_probe, azurerm_lb_rule, azurerm_network_security_group, +azurerm_network_security_rule, azurerm_role_assignment, azurerm_storage_account, +azurerm_storage_blob, azurerm_storage_container, azurerm_subnet, +azurerm_subnet_network_security_group_association, azurerm_user_assigned_identity, +azurerm_virtual_network, azurerm_storage_account_sas, azurerm_linux_virtual_machine, +azurerm_network_interface, azurerm_network_interface_backend_address_pool_association, +azurerm_dns_cname_record, azurerm_linux_virtual_machine, azurerm_network_security_rule` + +API update example: +A local variable should be defined, which merges the default tag and the user +defined Azure tags, which should be referred in the Azure resource APIs. +``` terraform +locals { + tags = merge( + { + "kubernetes.io_cluster.${var.cluster_id}" = "owned" + }, + var.azure_extra_tags, + ) +} + +resource "azurerm_resource_group" "main" { + tags = local.tags +} +``` + #### Create tags scenarios In the cases where a tag is specified in the `infrastructure.config.openshift.io` resource and 1) A tag with the same key is present in an Azure resource, the value will be replaced with @@ -379,8 +408,8 @@ On upgrade: On downgrade: - The status/spec field may remain populated, components may or may not continue - to tag newly created resources with the additional tags depending on whether or - not a given component still has logic to respect the status tags, after the downgrade. + to tag newly created resources with the additional tags depending on regardless of + whether given component still has logic to respect the spec tags, after the downgrade. ### Version Skew Strategy @@ -425,5 +454,6 @@ Motivations for having a dedicated cloud-infra-controller include but are not li - Few Azure resources such as disk, storage account, DNS zones and records name might not match either `kubernetes.io/cluster/:owned` or cluster name in the name tag and will result in tagging inconsistency, which will be minimal compared not to having - aformentioned dedicated controller for managing tags of Azure resources. -## Infrastructure Needed [optional] + aforementioned dedicated controller for managing tags of Azure resources. + +## Infrastructure Needed [optional] \ No newline at end of file From 4df6e50c30f8f39a264f32ce972053282455875d Mon Sep 17 00:00:00 2001 From: Bharath B Date: Wed, 20 Jul 2022 00:14:57 +0530 Subject: [PATCH 04/27] Update and rename azure_user_defined_tags.md to api-review/azure_user_defined_tags.md --- api-review/azure_user_defined_tags.md | 310 +++++++++++++++++ azure_user_defined_tags.md | 459 -------------------------- 2 files changed, 310 insertions(+), 459 deletions(-) create mode 100644 api-review/azure_user_defined_tags.md delete mode 100644 azure_user_defined_tags.md diff --git a/api-review/azure_user_defined_tags.md b/api-review/azure_user_defined_tags.md new file mode 100644 index 0000000000..7be6f7c003 --- /dev/null +++ b/api-review/azure_user_defined_tags.md @@ -0,0 +1,310 @@ +--- +title: apply-user-defined-tags-to-all-azure-resources-created-by-openShift +authors: + - "@bhb" +reviewers: + - @patrickdillon + - @sdodson + - @jerpeter1 + - @Miciah + - @sinnykumari + - @dmage + - @staebler + - @tkashem + - @tjungblu +approvers: + - @sdodson + - @jerpeter1 + - @bparees +api-approvers: + - @joelspeed +creation-date: 2022-07-12 +last-updated: 2022-07-12 +tracking-link: + - https://issues.redhat.com/browse/OCPPLAN-8155 +see-also: + - "enhancements/api-review/custom-tags-aws.md" +replaces: + - N/A +superseded-by: + - N/A +--- + +# Apply user defined tags to all Azure resources created by OpenShift + +## Summary + +This enhancement describes the proposal to allow an administrator of Openshift to +have the ability to apply user defined tags to those resources created by Openshift +in Azure. + +## Motivation + +Motivations include but are not limited to: + +- Allow admin, compliance, and security teams to keep track of assets and objects + created by OpenShift in Azure. + +### User Stories + +- As an openshift administrator, I want to have tags added to all resources created + in Azure by Openshift. +- As an openshift administrator, I want to restrict access granted to Openshift specific account. + +### Goals + +- The administrator or service (in the case of Managed OpenShift) installing OpenShift + can configure allowed number of user-defined tags in the Openshift installer generated + install config, which is referred and applied by installer and the in-cluster operators + on the the Azure resources during creation. +- Tags must be applied at creation time, in an atomic operation. It isn't acceptable + to create an object and, after some period of time, apply the tags post-creation. + +### Non-Goals + +- Management of resource tags post creation of cluster is out of scope. + +## Proposal + +A tag of the form `kubernetes.io/cluster/:owned` will be added to every +resource created by Openshift to enable administrator to differentiate the resources +created for Openshift cluster. An administrator is not allowed to add or modify the tag +having the prefix `kubernetes.io` or `openshift.io` in the name. + +New `userTags` field will be added to `platform.azure` of install-config for the user +to define the tags to be added to the resources created by installer and in-cluster operators. + +If `platform.azure.userTags` of install-config has any tag defined same will be added +to all the azure resources created by Openshift except, when the tag validation fails +due to any of below conditions +1. A tag name can have a maximum of 128 characters. + - Tag name has a limit of 512 characters for all resources except for + storage accounts, which has a limit of 128 characters and hence tag name + length is restricted to 128 characters on every resource required by Openshift. +2. A tag name cannot contain `<, >, %, &, \, ?, /, #, :, whitespace` characters and + should not start with a number. + - DNS zones, Traffic, Front Door resources does not support tag with spaces, + special/unicode characters or starting with number, hence these are added as + constraints on every other Azure resource required by Openshift as well. +3. A tag value has a limit of 256 characters. +4. A resource, resource-group or subscription, user can configure a maximum of 5 tags + through Openshift. + - Azure supports a maximum of 50 tags except for Automation, Content Delivery Network, + DNS resources which can have a maximum of 15 tags, hence restricting the number of + user defined tags to 5 and 10 for Openshift internal use for all resources created + by Openshift. + +All in-cluster operators that create Azure resources (Cluster Infrastructure ,Storage ,Node ,NetworkEdge , Internal Registry ,CCO) will apply these tags during resource creation. + +The userTags field is intended to be set at install time and is considered immutable. +Components that respect this field must only ever add tags that they retrieve from this +field to cloud resources, they must never remove tags from the existing underlying cloud +resource even if the tags are removed from this field(despite it being immutable). + +If the userTags field is changed post-install, there is no guarantee about how an +in-cluster operator will respond to the change. Some operators may reconcile the +change and change tags on the Azure resource. Some operators may ignore the change. +However, if tags are removed from userTags, the tag will not be removed from the +Azure resource. + +### Workflow Description + +- An Openshift administrator requests to add required tags to all Azure resources + created by Openshift by adding it in `.platform.azure.userTags` +- openshift installer validates the tags defined in `.platform.azure.userTags` and + adds these tags to all resources created during installation and also updates + `.spec.platformSpec.azure.resourceTags` of the `infrastructure.config.openshift.io` +- In cluster operators refers `.spec.platformSpec.azure.resourceTags` of the + `infrastructure.config.openshift.io` to add tags to resources created later. +- An Openshift administrator can modify existing tags or add new tags by updating + `.spec.platformSpec.azure.resourceTags` field in the `infrastructure.config.openshift.io`. + +#### Variation [optional] + +### API Extensions +Enhancement requires below modifications to the mentioned CRDs +- Add `userTags` field to `platform.azure` of the `installconfigs.install.openshift.io` +```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: installconfigs.install.openshift.io +spec: + versions: + - name: v1 + schema: + openAPIV3Schema: + properties: + platform: + properties: + azure: + properties: + userTags: + additionalProperties: + type: string + description: UserTags additional keys and values that the installer + will add as tags to all resources that it creates. Resources + created by the cluster itself may not include these tags. + type: object +``` + +- Add `resourceTags` field to `spec.platformSpec.azure` and `platformStatus.status.azure` + of the `infrastructure.config.openshift.io` +```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: infrastructures.config.openshift.io +spec: + versions: + - name: v1 + schema: + openAPIV3Schema: + properties: + status: + properties: + platformStatus: + properties: + azure: + properties: + resourceTags: + description: resourceTags is a list of additional tags to apply to Azure resources created for the cluster. See https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few, which is limited to 15. OpenShift reserves 10 tags for its internal use, and allows 5 tags for user configuration. + type: array + maxItems: 5 + items: + description: AzureResourceTag is a tag to apply to Azure resources created for the cluster. + type: object + required: + - key + - value + properties: + key: + description: key is the key of the tag + type: string + maxLength: 128 + minLength: 1 + pattern: ^[a-zA-Z][0-9A-Za-z_.=+-@]+$ + value: + description: value is the value of the tag. + type: string + maxLength: 256 + minLength: 1 + pattern: ^[0-9A-Za-z_.=+-@]+$ +``` + +### Implementation Details/Notes/Constraints [optional] +Add a new field `resourceTags` to `.status.platformStatus.azure` of the +`infrastructure.config.openshift.io` type. Tags included in the `resourceTags` field +will be applied to new resources created for the cluster by in-cluster operators. + +The `resourceTags` field in `spec.platformSpec.azure` will be populated by the +installer using the entries from `platform.azure.userTags` field of `install-config`. + +`infrastructure.config.openshift.io` field `status.platformStatus.azure` is immutable. + +All operators that create Azure resources (Cluster Infrastructure ,Storage ,Node , +NetworkEdge ,Internal Registry ,CCO) will apply these tags to all Azure resources they create. + +Below list of terraform Azure APIs to create resources should be updated to add user +defined tags and as well the openshift default tag in the installer component. +`azurerm_resource_group, azurerm_image, azurerm_lb, azurerm_lb_backend_address_pool, +azurerm_lb_probe, azurerm_lb_rule, azurerm_network_security_group, +azurerm_network_security_rule, azurerm_role_assignment, azurerm_storage_account, +azurerm_storage_blob, azurerm_storage_container, azurerm_subnet, +azurerm_subnet_network_security_group_association, azurerm_user_assigned_identity, +azurerm_virtual_network, azurerm_storage_account_sas, azurerm_linux_virtual_machine, +azurerm_network_interface, azurerm_network_interface_backend_address_pool_association, +azurerm_dns_cname_record, azurerm_linux_virtual_machine, azurerm_network_security_rule` + +API update example: +A local variable should be defined, which merges the default tag and the user +defined Azure tags, which should be referred in the Azure resource APIs. +``` terraform +locals { + tags = merge( + { + "kubernetes.io_cluster.${var.cluster_id}" = "owned" + }, + var.azure_extra_tags, + ) +} + +resource "azurerm_resource_group" "main" { + tags = local.tags +} +``` + +#### Caveats +1. User updates or removes Azure resource tags added by Openshift, using an external interface + will not be reconciled by the resource owning operator. +2. Updating tags of individual resources is not supported and any tag present in + `.status.platformStatus.azure.resourceTags` of `infrastructure.config.openshift.io/v1` resource + will result in addding tags to all Openshift managed Azure resources. + +### Risks and Mitigations + +### Drawbacks +- User-defined tags cannot be updated on an Azure resource which is not managed by an + operator. In this proposal, the changes proposed and developed will be part of + openshift-* namespace. External operators are not in scope. + User-defined tags can be updated on the following Azure resources. + 1. Virtual machine resources created for master and worker nodes. + 2. Storage account + 3. NetworkEdge + 4. Internal Registry + 5. CCO + +- OpenShift is bound to have the common limitation for all Azure resources created + by it and constrains other resources with the least matching limit as below + 1. Tag names cannot have `microsoft`, `azure`, `windows` prefixes which are + reserved for Azure use. + 2. An Azure storage account has a limit of 128 characters for the tag name. + 3. An Azure DNS zone or Traffic or Front Door resource tag name cannot have spaces, + special/unicode characters or start with a number. + 4. An Azure Automation or Content Delivery Network or DNS resource can have a + maximum of 15 tags. + +- Administrator will have to manually perform tags pertaining actions for + 1. removing the undesired tags from the required resources. + 2. update tag values of the required resources. + 3. update tags of the resources which are not managed by an operator. + 4. update tags of the resources for which update logic is not supported by an operator. + +## Design Details + +### Open Questions + +### Test Plan + +### Graduation Criteria + +#### Dev Preview -> Tech Preview + +#### Tech Preview -> GA + +#### Removing a deprecated feature + +### Upgrade / Downgrade Strategy + +On upgrade: +- Cluster operators that update the tags of Azure resources created for cluster + should refer the new fields and take action. + +On downgrade: +- The status/spec field may remain populated, components may or may not continue + to tag newly created resources with the additional tags depending on regardless of + whether given component still has logic to respect the spec tags, after the downgrade. + +### Version Skew Strategy + +### Operational Aspects of API Extensions + +#### Failure Modes + +#### Support Procedures + +## Implementation History + +## Alternatives + +## Infrastructure Needed [optional] diff --git a/azure_user_defined_tags.md b/azure_user_defined_tags.md deleted file mode 100644 index e6032ae04b..0000000000 --- a/azure_user_defined_tags.md +++ /dev/null @@ -1,459 +0,0 @@ ---- -title: apply-user-defined-tags-to-all-azure-resources-created-by-openShift -authors: - - "@bhb" -reviewers: - - -approvers: - - -api-approvers: - - -creation-date: 2022-07-12 -last-updated: 2022-07-12 -tracking-link: - - https://issues.redhat.com/browse/RFE-2017 - - https://issues.redhat.com/browse/CFE-492 - - https://issues.redhat.com/browse/CFEPLAN-59 -see-also: - - -replaces: - - -superseded-by: - - ---- - -# Apply user defined tags to all Azure resources created by OpenShift - -## Summary - -This enhancement describes the proposal to allow an administrator of Openshift to -have the ability to apply user defined tags to those resources created by Openshift -in Azure. - -## Motivation - -Motivations include but are not limited to: - -- Allow admin, compliance, and security teams to keep track of assets and objects - created by OpenShift, both at install time and during continuous operation (Day 2) - -### User Stories - -- As an openshift administrator, I want to have tags added to all resources created - in Azure by Openshift on Day 1. -- As an openshift administrator, I want to restrict access granted to Openshift specific account. -- As an openshift administrator, I should be able to create new tags or modify existing tags. - -### Goals - -- The administrator or service (in the case of Managed OpenShift) installing OpenShift - can pass an arbitrary list of user-defined tags to the OpenShift installer, and - the installer and all the bootstrapped components will apply these tags to all - resources created in Azure, for the life of the cluster, and where supported by Azure. -- Tags must be applied at creation time, in an atomic operation. It isn't acceptable - to create an object and, after some period of time, apply the tags post-creation. - -### Non-Goals - -- To reduce initial scope, support for deleting the tags is not supported. - -## Proposal - -A tag of the form `kubernetes.io/cluster/:owned` will be added to every -resource created by Openshift to enable administrator to differentiate the resources -created for Openshift cluster. An administrator is not allowed to add or modify the tag -having the prefix `kubernetes.io` in the name. - -New `userTags` field will be added to `platform.azure` of install-config for the user -to define the tags to be added to the resources created by installer and in-cluster operators. - -If `platform.azure.userTags` of install-config has any tag defined same will be added -to all the azure resources created by Openshift except, when the tag validation fails -due to any of below conditions
-1. A tag name can have a maximum of 128 characters
- (Note: Tag name has a limit of 512 characters for all resources except for - storage accounts, which has a limit of 128 characters and hence tag name - length is restricted to 128 characters for every resource required by Openshift)
-2. A tag value has a limit of 256 characters.
-3. A tag name cannot contain `<, >, %, &, \, ?, /, #, :, whitespace` characters and - should not start with a number.
- (Note: DNS zones, Traffic, Front Door resources does not support tag with spaces, - special/unicode characters or starting with number, hence these are added as - constraints to every other azure resource required by Openshift as well.)
-4. A resource, resource-group or subscription can have a maximum of 10 tags.
- (Note: Azure supports a maximum of 50 tags except for Automation, - Content Delivery Network, DNS resources which can have a maximum of 15 tags, hence - restricting the number of tags to 10 for all resources created by Openshift, with 5 - spared for administrator's use)
- -Add a new field `resourceTags` to `.spec.platformSpec.azure` of the -`infrastructure.config.openshift.io` type. Tags included in the `resourceTags` field -will be applied to new resources created for the cluster. - -Add a new field `resourceTags` to `.status.platformStatus.azure` of the -`infrastructure.config.openshift.io` type. `resourceTags` will have the information -on the resources for which tag operation failed, updated by the respective in-cluster operator. - -All operators that create Azure resources (Cluster Infrastructure ,Storage ,Node ,NetworkEdge , -Internal Registry ,CCO) will apply these tags to all Azure resources they create. - -`resourceTags` that are specified in the infrastructure resource will merge with tags -specified in an Azure resource. In the case where `resourceTags` specified in the -infrastructure resource and there is a tag with the same name specified in an Azure -resource, the value from the infrastructure resource will take precedence and be updated. - -The userTags field is intended to be set at install time and is considered immutable. -Components that respect this field must only ever add tags that they retrieve from this -field to cloud resources, they must never remove tags from the existing underlying cloud -resource even if the tags are removed from this field(despite it being immutable). - -If the userTags field is changed post-install, there is no guarantee about how an -in-cluster operator will respond to the change. Some operators may reconcile the -change and change tags on the Azure resource. Some operators may ignore the change. -However, if tags are removed from userTags, the tag will not be removed from the -Azure resource. - -### Workflow Description - -- An Openshift administrator requests to add required tags to all Azure resources - created by Openshift by adding it in `.platform.azure.userTags` -- openshift installer validates the tags defined in `.platform.azure.userTags` and - adds these tags to all resources created during installation and also updates - `.spec.platformSpec.azure.resourceTags` of the `infrastructure.config.openshift.io` -- In cluster operators refers `.spec.platformSpec.azure.resourceTags` of the - `infrastructure.config.openshift.io` to add tags to resources created later. -- An Openshift administrator can modify existing tags or add new tags by updating - `.spec.platformSpec.azure.resourceTags` field in the `infrastructure.config.openshift.io`. - -#### Variation [optional] - -### API Extensions -Enhancement requires below modifications to the mentioned CRDs -- Add `userTags` field to `platform.azure` of the `installconfigs.install.openshift.io` -```yaml -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - name: installconfigs.install.openshift.io -spec: - versions: - - name: v1 - schema: - openAPIV3Schema: - description: InstallConfig is the configuration for an OpenShift install. - properties: - platform: - description: Platform is configuration for machine pool specific - to the platform. - properties: - azure: - description: Azure is the configuration used when installing - on Azure. - properties: - userTags: - additionalProperties: - type: string - description: UserTags additional keys and values that the installer - will add as tags to all resources that it creates. Resources - created by the cluster itself may not include these tags. - type: object -``` - -- Add `resourceTags` field to `spec.platformSpec.azure` and `platformStatus.status.azure` - of the `infrastructure.config.openshift.io` -```yaml -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - name: infrastructures.config.openshift.io -spec: - versions: - - name: v1 - schema: - openAPIV3Schema: - description: "Infrastructure holds cluster-wide information about Infrastructure. The canonical name is `cluster` \n Compatibility level 1: Stable within a major release for a minimum of 12 months or 3 minor releases (whichever is longer)." - properties: - spec: - description: spec holds user settable values for configuration - properties: - platformSpec: - description: platformSpec holds desired information specific to the underlying infrastructure provider. - properties: - azure: - description: Azure contains settings specific to the Azure infrastructure provider. - properties: - resourceTags: - description: resourceTags is a list of additional tags to apply to Azure resources created for the cluster. See https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few. OpenShift reserves 10 tags for its use, leaving 40 tags available for the user. - type: array - maxItems: 10 - items: - description: AzureResourceTag is a tag to apply to Azure resources created for the cluster. - type: object - required: - - key - - value - properties: - key: - description: key is the key of the tag - type: string - maxLength: 128 - minLength: 1 - pattern: ^[a-zA-Z][0-9A-Za-z_.=+-@]+$ - value: - description: value is the value of the tag. - type: string - maxLength: 256 - minLength: 1 - pattern: ^[0-9A-Za-z_.=+-@]+$ - status: - description: status holds observed values from the cluster. They may not be overridden. - properties: - platformStatus: - description: platformStatus holds status information specific to the underlying infrastructure provider. - properties: - azure: - description: Azure contains settings specific to the Azure infrastructure provider. - properties: - resourceTags: - description: resourceTags is a list of Azure resources for which tag update got failed. It contains the Azure resource name and the encountered error. - type: array - items: - description: Azure resource is the name of the resource created for the cluster. - type: object - required: - - name - - error - properties: - name: - description: name is the name of the Azure resource. - type: string - error: - description: error is the reason for the tag update failure. - type: string -``` - -### Implementation Details/Notes/Constraints [optional] - -What are the caveats to the implementation? What are some important details that -didn't come across above. Go in to as much detail as necessary here. This might -be a good place to talk about core concepts and how they relate. - -### Risks and Mitigations - -### Drawbacks -- User-defined tags cannot be updated on an Azure resource which is not managed by an - operator. In this proposal, the changes proposed and developed will be part of - openshift-* namespace. External operators are not in scope. - User-defined tags can be updated on the following Azure resources. - 1. Virtual machine resources created for master and worker nodes. - 2. Storage account - 3. NetworkEdge - 4. Internal Registry - 5. CCO - -- OpenShift is bound to have the common limitation for all Azure resources created - by it and constrains other resources with the least matching limit as below - 1. Tag names cannot have `microsoft`, `azure`, `windows` prefixes which are - reserved for Azure use. - 2. An Azure storage account has a limit of 128 characters for the tag name. - 3. An Azure DNS zone or Traffic or Front Door resource tag name cannot have spaces, - special/unicode characters or start with a number. - 4. An Azure Automation or Content Delivery Network or DNS resource can have a - maximum of 15 tags. - -- Administrator will have to manually perform tags pertaining actions for - 1. removing the undesired tags from the required resources. - 2. update tags of the resources which are not managed by an operator. - 3. update tags of the resources for which update logic is not supported by an operator. - -## Design Details -The `resourceTags` field in `spec.platformSpec.azure` will be populated by the -installer using the entries from `userTags` field. - -`infrastructure.config.openshift.io` fields `spec.platformSpec.azure` is a mutable -and `status.platformStatus.azure` is immutable. - -All operators that create Azure resources (Cluster Infrastructure ,Storage ,Node , -NetworkEdge ,Internal Registry ,CCO) will apply these tags to all Azure resources they create. - -Operators should create an event to indicate the status of tag add/modify request and -as well update the `status.platformSpec.azure` field of `infrastructure.config.openshift.io` -with required details in case of failure. - -Below list of terraform Azure APIs to create resources should be updated to add user -defined tags and as well the openshift default tag in the installer component. -`azurerm_resource_group, azurerm_image, azurerm_lb, azurerm_lb_backend_address_pool, -azurerm_lb_probe, azurerm_lb_rule, azurerm_network_security_group, -azurerm_network_security_rule, azurerm_role_assignment, azurerm_storage_account, -azurerm_storage_blob, azurerm_storage_container, azurerm_subnet, -azurerm_subnet_network_security_group_association, azurerm_user_assigned_identity, -azurerm_virtual_network, azurerm_storage_account_sas, azurerm_linux_virtual_machine, -azurerm_network_interface, azurerm_network_interface_backend_address_pool_association, -azurerm_dns_cname_record, azurerm_linux_virtual_machine, azurerm_network_security_rule` - -API update example: -A local variable should be defined, which merges the default tag and the user -defined Azure tags, which should be referred in the Azure resource APIs. -``` terraform -locals { - tags = merge( - { - "kubernetes.io_cluster.${var.cluster_id}" = "owned" - }, - var.azure_extra_tags, - ) -} - -resource "azurerm_resource_group" "main" { - tags = local.tags -} -``` - -#### Create tags scenarios -In the cases where a tag is specified in the `infrastructure.config.openshift.io` resource and -1) A tag with the same key is present in an Azure resource, the value will be replaced with - `infrastructure.config.openshift.io` resource value - - For example,
- Azure resource as a tag `key_infra = value_azure`
- New tag request = `spec.platformSpec.azure.resourceTags` has `key_infra = value_infra`
- Action = Azure resource tag is updated to `key_infra = value_infra`
- Event action = An event is generated to notify user about the request - status (success/failure) to update tags for the Azure resource. - -2) A tag with same key is not present and is a new tag to be added for an Aure - resource, but maximum tag limit is reached or any other error encountered - should be notified to user. - - For example,
- New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_infra`
- Action = A new tag to be added for an Azure resource.
- Final tag set to Azure resource = `key_infra = value_infra`
- Event action = An event is generated to notify user about the request - status (success/failure) to create tags for the Azure resource. - -#### Update tags scenarios -Users can update the user-defined tags by editing `.spec.platformSpec.azure` of the -`infrastructure.config.openshift.io` resource. On update of `resourceTags`, -Azure resource is not created or restarted. - -In the case where a tag is updated in the `infrastructure.config.openshift.io` resource and -1) A tag with the same key and value is present in an Azure resource, no update is made. - - For example,
- Existing tag for Azure resource = `key_infra = value_update`
- New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_update`
- Action = There is no update for Azure resource.
- Final tag set to Azure resource = `key_infra = value_update` - -2) A tag with the same key but different value is present in an Azure resource, the Azure - resource will be updated with the new value. - - For example,
- Existing tag for Azure resource = `key_infra = value`
- New tag request = `.spec.platformSpec.azure.resourceTags` has `key_infra = value_new`
- Action = Existing tag for Azure resource is updated to reflect new value.
- Final tag set to Azure resource = `key_infra = value_new`
- Event action = An event is generated to notify user about the request - status (success/failure) to update tags for the Azure resource. - -#### Caveats -1) User updates Azure resource's tag using an external interface which is present in - `.spec.platformSpec.azure.resourceTags`, tag will be reconciled by the owning operator - with the value from the `.spec.platformSpec.azure.resourceTags`. Reconciliation does not - happen set immediately for Azure resource by the owning operator. There is eventual - consistency maintained by the owning operator. The time taken to reconcile the modified - tag on Azure resource to desired value vary across owning operators. - - For example,
- Edited existing tag using external tool for Azure resource = `key_infra = value_tool`
- Previous tag request = `.spec.platformSpec.azure.resourceTags` or resource has `key_infra = value`
- Action = Update existing tag with value from `.spec.platformSpec.azure.resourceTags`.
- Final tag set to Azure resource = `key_infra1 = value`
- Event action = An event is generated to notify user about the request - status (success/failure) to update tags for the Azure resource. - -2) User removes the user-defined tag from `.spec.platformSpec.azure.resourceTags` or Azure resource. - The user-defined tag which is removed from spec will not be reconciled or managed by operators. - User can update tag using an external interface. The user-defined tag will not be modified - by the operator. - - For example,
- Existing tag for Azure resource = `key_infra = value`
- New tag request = `.spec.platformSpec.azure.resourceTags` has no user-defined tag with key `key_infra`
- Action = No change in tags for the Azure resource.
- Final tag set to Azure resource = `key_infra1 = value`
- -3) Updating tags of individual resources is not supported and any tag present in - `.spec.platformSpec.azure.resourceTags` of `infrastructure.config.openshift.io/v1` resource - will result in updating tags of all Openshift managed Azure resources. - -### Open Questions - -### Test Plan - -### Graduation Criteria - -#### Dev Preview -> Tech Preview - -#### Tech Preview -> GA - -#### Removing a deprecated feature - -### Upgrade / Downgrade Strategy - -On upgrade: -- Cluster operators that update the tags of Azure resources created for cluster - should refer the new fields and take action. - -On downgrade: -- The status/spec field may remain populated, components may or may not continue - to tag newly created resources with the additional tags depending on regardless of - whether given component still has logic to respect the spec tags, after the downgrade. - -### Version Skew Strategy - -### Operational Aspects of API Extensions - -#### Failure Modes - -#### Support Procedures - -## Implementation History - -## Alternatives -Alternate or extended proposal is to have a dedicated controller for managing -the cloud infrastructure as desired by the user by supporting but not limited -to adding or modifying tags of the cloud resources created by Openshift. - -### Motivation -Motivations for having a dedicated cloud-infra-controller include but are not limited to -- A single controller which adds/modifies tags of the all resources created by - Openshift and updates the status of the operation in `infrastructure.config.openshift.io` - instead of multiple controllers doing so. -- Having single controller will solve the issue of tags not getting added to those resources - created by installer and not being managed by any operator during tag update or cluster - upgrade scenario(from feature unsupported version). - -### Design Details -- The dedicated controller will watch for changes in the `infrastructure.config.openshift.io` - resource and acts only when there is a change to `spec.platformSpec.azure` field. -- Controller queries for all the resources which has the tag - `kubernetes.io/cluster/:owned` and updates each resource with the new - requested changes. -- In case cluster is upgraded from a release without Azure tag implementation (query for - resources with `kubernetes.io/cluster/:owned` tag yields just resource group - type), controller queries for the resources having cluster name in the name tag and - updates each matching resource with the new requested changes. -- In-cluster operators should not watch for changes to `spec.platformSpec.azure` field of - `infrastructure.config.openshift.io` and should continue with current functionality of - creating requested resources, but include additional functionality to add user defined tags - and the `kubernetes.io/cluster/:owned` default tag to created resource. - -### Caveats -- Few Azure resources such as disk, storage account, DNS zones and records name might not - match either `kubernetes.io/cluster/:owned` or cluster name in the name tag - and will result in tagging inconsistency, which will be minimal compared not to having - aforementioned dedicated controller for managing tags of Azure resources. - -## Infrastructure Needed [optional] \ No newline at end of file From cf49a7aa7a90b701ce28661894045c1028760d57 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 21 Jul 2022 10:57:58 +0530 Subject: [PATCH 05/27] updated path --- .../api-review}/azure_user_defined_tags.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {api-review => enhancements/api-review}/azure_user_defined_tags.md (100%) diff --git a/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md similarity index 100% rename from api-review/azure_user_defined_tags.md rename to enhancements/api-review/azure_user_defined_tags.md From ff18c761dc25e1e0fa67bd93f0cdedc15ba34900 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 18 Aug 2022 11:13:15 +0530 Subject: [PATCH 06/27] Update azure_user_defined_tags.md --- .../api-review/azure_user_defined_tags.md | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 7be6f7c003..334b6de082 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -55,7 +55,7 @@ Motivations include but are not limited to: - The administrator or service (in the case of Managed OpenShift) installing OpenShift can configure allowed number of user-defined tags in the Openshift installer generated - install config, which is referred and applied by installer and the in-cluster operators + install config, which is referred and applied by the installer and the in-cluster operators on the the Azure resources during creation. - Tags must be applied at creation time, in an atomic operation. It isn't acceptable to create an object and, after some period of time, apply the tags post-creation. @@ -82,17 +82,17 @@ due to any of below conditions storage accounts, which has a limit of 128 characters and hence tag name length is restricted to 128 characters on every resource required by Openshift. 2. A tag name cannot contain `<, >, %, &, \, ?, /, #, :, whitespace` characters and - should not start with a number. + must not start with a number. - DNS zones, Traffic, Front Door resources does not support tag with spaces, special/unicode characters or starting with number, hence these are added as constraints on every other Azure resource required by Openshift as well. -3. A tag value has a limit of 256 characters. +3. A tag value can have a maximum of 256 characters. 4. A resource, resource-group or subscription, user can configure a maximum of 5 tags through Openshift. - Azure supports a maximum of 50 tags except for Automation, Content Delivery Network, DNS resources which can have a maximum of 15 tags, hence restricting the number of - user defined tags to 5 and 10 for Openshift internal use for all resources created - by Openshift. + user defined tags to 5 and 10 for Openshift's internal use, for all the resources + created by Openshift. All in-cluster operators that create Azure resources (Cluster Infrastructure ,Storage ,Node ,NetworkEdge , Internal Registry ,CCO) will apply these tags during resource creation. @@ -113,11 +113,9 @@ Azure resource. created by Openshift by adding it in `.platform.azure.userTags` - openshift installer validates the tags defined in `.platform.azure.userTags` and adds these tags to all resources created during installation and also updates - `.spec.platformSpec.azure.resourceTags` of the `infrastructure.config.openshift.io` -- In cluster operators refers `.spec.platformSpec.azure.resourceTags` of the + `.status.platformStatus.azure.resourceTags` of the `infrastructure.config.openshift.io` +- In cluster operators refers `.sttaus.platformSTatus.azure.resourceTags` of the `infrastructure.config.openshift.io` to add tags to resources created later. -- An Openshift administrator can modify existing tags or add new tags by updating - `.spec.platformSpec.azure.resourceTags` field in the `infrastructure.config.openshift.io`. #### Variation [optional] @@ -197,10 +195,10 @@ Add a new field `resourceTags` to `.status.platformStatus.azure` of the `infrastructure.config.openshift.io` type. Tags included in the `resourceTags` field will be applied to new resources created for the cluster by in-cluster operators. -The `resourceTags` field in `spec.platformSpec.azure` will be populated by the -installer using the entries from `platform.azure.userTags` field of `install-config`. +The `resourceTags` field in `status.platformStatus.azure` of `infrastructure.config.openshift.io` +will be populated by the installer using the entries from `platform.azure.userTags` field of `install-config`. -`infrastructure.config.openshift.io` field `status.platformStatus.azure` is immutable. +`status.platformStatus.azure` of `infrastructure.config.openshift.io` is immutable. All operators that create Azure resources (Cluster Infrastructure ,Storage ,Node , NetworkEdge ,Internal Registry ,CCO) will apply these tags to all Azure resources they create. @@ -255,7 +253,7 @@ resource "azurerm_resource_group" "main" { 5. CCO - OpenShift is bound to have the common limitation for all Azure resources created - by it and constrains other resources with the least matching limit as below + by it and constraints other resources with the least matching limit as below 1. Tag names cannot have `microsoft`, `azure`, `windows` prefixes which are reserved for Azure use. 2. An Azure storage account has a limit of 128 characters for the tag name. @@ -264,7 +262,7 @@ resource "azurerm_resource_group" "main" { 4. An Azure Automation or Content Delivery Network or DNS resource can have a maximum of 15 tags. -- Administrator will have to manually perform tags pertaining actions for +- Administrator will have to manually perform below tags pertaining actions 1. removing the undesired tags from the required resources. 2. update tag values of the required resources. 3. update tags of the resources which are not managed by an operator. From 4f7e1ac565c744cd1fe45aeb2666275c94034634 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 18 Aug 2022 11:32:35 +0530 Subject: [PATCH 07/27] Update azure_user_defined_tags.md --- enhancements/api-review/azure_user_defined_tags.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 334b6de082..a17c22256d 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -205,14 +205,7 @@ NetworkEdge ,Internal Registry ,CCO) will apply these tags to all Azure resource Below list of terraform Azure APIs to create resources should be updated to add user defined tags and as well the openshift default tag in the installer component. -`azurerm_resource_group, azurerm_image, azurerm_lb, azurerm_lb_backend_address_pool, -azurerm_lb_probe, azurerm_lb_rule, azurerm_network_security_group, -azurerm_network_security_rule, azurerm_role_assignment, azurerm_storage_account, -azurerm_storage_blob, azurerm_storage_container, azurerm_subnet, -azurerm_subnet_network_security_group_association, azurerm_user_assigned_identity, -azurerm_virtual_network, azurerm_storage_account_sas, azurerm_linux_virtual_machine, -azurerm_network_interface, azurerm_network_interface_backend_address_pool_association, -azurerm_dns_cname_record, azurerm_linux_virtual_machine, azurerm_network_security_rule` +`azurerm_resource_group, azurerm_image, azurerm_lb, azurerm_network_security_group, azurerm_storage_account, azurerm_user_assigned_identity, azurerm_virtual_network, azurerm_linux_virtual_machine, azurerm_network_interface, azurerm_dns_cname_record` API update example: A local variable should be defined, which merges the default tag and the user From fc9392fad109727a984d21a5e544b5f4e6fd77b1 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 18 Aug 2022 12:12:03 +0530 Subject: [PATCH 08/27] Update azure_user_defined_tags.md --- enhancements/api-review/azure_user_defined_tags.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index a17c22256d..690e40121c 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -146,14 +146,14 @@ spec: type: object ``` -- Add `resourceTags` field to `spec.platformSpec.azure` and `platformStatus.status.azure` +- Add `resourceTags` field to `platformStatus.status.azure` of the `infrastructure.config.openshift.io` ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: infrastructures.config.openshift.io -spec: +status: versions: - name: v1 schema: @@ -282,9 +282,9 @@ On upgrade: should refer the new fields and take action. On downgrade: -- The status/spec field may remain populated, components may or may not continue +- The status field may remain populated, components may or may not continue to tag newly created resources with the additional tags depending on regardless of - whether given component still has logic to respect the spec tags, after the downgrade. + whether given component still has logic to respect the status tags, after the downgrade. ### Version Skew Strategy From 84c70221e114004da7ebdd0c9268de1a0afceae5 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Mon, 22 Aug 2022 10:56:49 +0530 Subject: [PATCH 09/27] reviewers name in quotes --- .../api-review/azure_user_defined_tags.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 690e40121c..7bd9690903 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -3,21 +3,21 @@ title: apply-user-defined-tags-to-all-azure-resources-created-by-openShift authors: - "@bhb" reviewers: - - @patrickdillon - - @sdodson - - @jerpeter1 - - @Miciah - - @sinnykumari - - @dmage - - @staebler - - @tkashem - - @tjungblu + - "@patrickdillon" + - "@sdodson" + - "@jerpeter1" + - "@Miciah" + - "@sinnykumari" + - "@dmage" + - "@staebler" + - "@tkashem" + - "@tjungblu" approvers: - - @sdodson - - @jerpeter1 - - @bparees + - "@sdodson" + - "@jerpeter1" + - "@bparees" api-approvers: - - @joelspeed + - "@joelspeed" creation-date: 2022-07-12 last-updated: 2022-07-12 tracking-link: From acb8eddb9f5e942f511516b9effe037c18fae0f6 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Tue, 6 Sep 2022 12:08:51 +0530 Subject: [PATCH 10/27] Update azure_user_defined_tags.md --- .../api-review/azure_user_defined_tags.md | 53 ++++++++++++------- 1 file changed, 34 insertions(+), 19 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 7bd9690903..1466429ec2 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -1,27 +1,33 @@ --- -title: apply-user-defined-tags-to-all-azure-resources-created-by-openShift +title: azure_user_defined_tags authors: - "@bhb" reviewers: - "@patrickdillon" - "@sdodson" - - "@jerpeter1" - - "@Miciah" - - "@sinnykumari" + - "@jhixson74" + - "@JoelSpeed" + - "@dhellmann" + - "@jewzaam" + - "@adambkaplan" + - "@abhat" - "@dmage" + - "@Miciah" + - "@abutcher" - "@staebler" - - "@tkashem" - - "@tjungblu" + - "@jerpeter1" approvers: - "@sdodson" - "@jerpeter1" - "@bparees" api-approvers: - "@joelspeed" + - "@bparees" creation-date: 2022-07-12 last-updated: 2022-07-12 tracking-link: - https://issues.redhat.com/browse/OCPPLAN-8155 + - https://issues.redhat.com/browse/CORS-2249 see-also: - "enhancements/api-review/custom-tags-aws.md" replaces: @@ -49,20 +55,21 @@ Motivations include but are not limited to: - As an openshift administrator, I want to have tags added to all resources created in Azure by Openshift. -- As an openshift administrator, I want to restrict access granted to Openshift specific account. +- As an openshift administrator, I want to restrict access granted to Openshift + specific account. ### Goals - The administrator or service (in the case of Managed OpenShift) installing OpenShift can configure allowed number of user-defined tags in the Openshift installer generated install config, which is referred and applied by the installer and the in-cluster operators - on the the Azure resources during creation. + on the the Azure resources during cluster creation. - Tags must be applied at creation time, in an atomic operation. It isn't acceptable - to create an object and, after some period of time, apply the tags post-creation. + to create an object and to apply tags post cluster creation. ### Non-Goals -- Management of resource tags post creation of cluster is out of scope. +- Management(update/delete) of resource tags post creation of cluster is out of scope. ## Proposal @@ -75,8 +82,8 @@ New `userTags` field will be added to `platform.azure` of install-config for the to define the tags to be added to the resources created by installer and in-cluster operators. If `platform.azure.userTags` of install-config has any tag defined same will be added -to all the azure resources created by Openshift except, when the tag validation fails -due to any of below conditions +to all the azure resources created by Openshift except, when the tag validation fail +to meet any of the below conditions 1. A tag name can have a maximum of 128 characters. - Tag name has a limit of 512 characters for all resources except for storage accounts, which has a limit of 128 characters and hence tag name @@ -114,7 +121,7 @@ Azure resource. - openshift installer validates the tags defined in `.platform.azure.userTags` and adds these tags to all resources created during installation and also updates `.status.platformStatus.azure.resourceTags` of the `infrastructure.config.openshift.io` -- In cluster operators refers `.sttaus.platformSTatus.azure.resourceTags` of the +- In cluster operators refers `.status.platformStatus.azure.resourceTags` of the `infrastructure.config.openshift.io` to add tags to resources created later. #### Variation [optional] @@ -146,14 +153,14 @@ spec: type: object ``` -- Add `resourceTags` field to `platformStatus.status.azure` +- Add `resourceTags` field to `status.platformStatus.azure` of the `infrastructure.config.openshift.io` ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: infrastructures.config.openshift.io -status: +spec: versions: - name: v1 schema: @@ -166,7 +173,7 @@ status: azure: properties: resourceTags: - description: resourceTags is a list of additional tags to apply to Azure resources created for the cluster. See https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few, which is limited to 15. OpenShift reserves 10 tags for its internal use, and allows 5 tags for user configuration. + description: resourceTags is a list of additional tags to apply to Azure resources created for the cluster. See https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few, which have limitation of 15 tags. OpenShift reserves 10 tags for its internal use, and allows 5 tags for user configuration. type: array maxItems: 5 items: @@ -193,15 +200,23 @@ status: ### Implementation Details/Notes/Constraints [optional] Add a new field `resourceTags` to `.status.platformStatus.azure` of the `infrastructure.config.openshift.io` type. Tags included in the `resourceTags` field -will be applied to new resources created for the cluster by in-cluster operators. +will be applied to new resources created for the cluster by the in-cluster operators. The `resourceTags` field in `status.platformStatus.azure` of `infrastructure.config.openshift.io` will be populated by the installer using the entries from `platform.azure.userTags` field of `install-config`. `status.platformStatus.azure` of `infrastructure.config.openshift.io` is immutable. -All operators that create Azure resources (Cluster Infrastructure ,Storage ,Node , -NetworkEdge ,Internal Registry ,CCO) will apply these tags to all Azure resources they create. +All operators that create Azure resources will apply these tags to all Azure +resources they create. + +| Operator | Resources created by the operator | +| -------- | ----------------------------- | +| cloud-network-config-controller | Private IP address | +| cluster-image-registry-operator | Storage Account | +| cluster-ingress-operator | Load Balancer, DNS records | +| cloud-credential-operator | IAM roles and policies | +| machine-api-provider-azure | Application Security Group, Availability Set, Group, Load Balancer, Public IP Address, Route, Network Security Group, Virtual Machine Extension, Virtual Interface, Virtual Machine, Virtual Network. | Below list of terraform Azure APIs to create resources should be updated to add user defined tags and as well the openshift default tag in the installer component. From 0c81a8445a0219e01dad311821db77116706a235 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Tue, 6 Sep 2022 12:37:26 +0530 Subject: [PATCH 11/27] fixed ci lint reported error --- enhancements/api-review/azure_user_defined_tags.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 1466429ec2..7f2b8740e4 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -173,7 +173,13 @@ spec: azure: properties: resourceTags: - description: resourceTags is a list of additional tags to apply to Azure resources created for the cluster. See https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few, which have limitation of 15 tags. OpenShift reserves 10 tags for its internal use, and allows 5 tags for user configuration. + description: resourceTags is a list of additional tags to apply to Azure + resources created for the cluster. See + https://docs.microsoft.com/en-us/rest/api/resources/tags for information + on tagging Azure resources. Azure supports a maximum of 50 tags per + resource except for few, which have limitation of 15 tags. OpenShift + reserves 10 tags for its internal use, and allows 5 tags + for user configuration. type: array maxItems: 5 items: From 742a1fc5f1440d7d0dd2ade7f5f52b6288cac692 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Wed, 7 Sep 2022 15:15:14 +0530 Subject: [PATCH 12/27] Update enhancements/api-review/azure_user_defined_tags.md Co-authored-by: Ben Parees --- enhancements/api-review/azure_user_defined_tags.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 7f2b8740e4..eac61478b2 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -251,7 +251,7 @@ resource "azurerm_resource_group" "main" { will not be reconciled by the resource owning operator. 2. Updating tags of individual resources is not supported and any tag present in `.status.platformStatus.azure.resourceTags` of `infrastructure.config.openshift.io/v1` resource - will result in addding tags to all Openshift managed Azure resources. + will result in adding tags to all Openshift managed Azure resources. ### Risks and Mitigations From a26b915dbf5ede555b6051b3d4dd74e5eba7d775 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 8 Sep 2022 16:53:12 +0530 Subject: [PATCH 13/27] incorporated comments --- .../api-review/azure_user_defined_tags.md | 40 +++++++++---------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index eac61478b2..8235469b8c 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -3,26 +3,17 @@ title: azure_user_defined_tags authors: - "@bhb" reviewers: - - "@patrickdillon" - - "@sdodson" - - "@jhixson74" - - "@JoelSpeed" - - "@dhellmann" - - "@jewzaam" - - "@adambkaplan" - - "@abhat" - - "@dmage" - - "@Miciah" - - "@abutcher" - - "@staebler" - - "@jerpeter1" + - "@patrickdillon" ## reviewer for installer component + - "@JoelSpeed" ## reviewer for machine-api-provider-azure component + - "@dmage" ## reviewer for cluster-image-registry-operator component + - "@Miciah" ## reviewer for cluster-ingress-operator component + - "@akhil-rane" ## reviewer for cloud-credential-operator component + - TBD ## reviewer for api component + - TBD ## reviewer for cloud-network-config-controller component approvers: - - "@sdodson" - - "@jerpeter1" - - "@bparees" + - "@jerpeter1" ## approver for CFE api-approvers: - - "@joelspeed" - - "@bparees" + - TBD creation-date: 2022-07-12 last-updated: 2022-07-12 tracking-link: @@ -76,7 +67,8 @@ Motivations include but are not limited to: A tag of the form `kubernetes.io/cluster/:owned` will be added to every resource created by Openshift to enable administrator to differentiate the resources created for Openshift cluster. An administrator is not allowed to add or modify the tag -having the prefix `kubernetes.io` or `openshift.io` in the name. +having the prefix `kubernetes.io` or `openshift.io` in the name. The same tag can be +found applied to other cloud platform resources which supports tagging for ex: AWS. New `userTags` field will be added to `platform.azure` of install-config for the user to define the tags to be added to the resources created by installer and in-cluster operators. @@ -214,7 +206,10 @@ will be populated by the installer using the entries from `platform.azure.userTa `status.platformStatus.azure` of `infrastructure.config.openshift.io` is immutable. All operators that create Azure resources will apply these tags to all Azure -resources they create. +resources they create. List of in-cluster operators managing cloud resources +could vary across platform types, example for AWS there are additional operators +like aws-ebs-csi-driver-operator, aws-efs-csi-driver-operator to manage specific +resources. | Operator | Resources created by the operator | | -------- | ----------------------------- | @@ -300,7 +295,10 @@ resource "azurerm_resource_group" "main" { On upgrade: - Cluster operators that update the tags of Azure resources created for cluster - should refer the new fields and take action. + should refer the new fields and take action. Any new resource created post-upgrade + and the operators managing the resource will add the user defined tags to the + resource. But the same does not apply to already existing resources, components may + or may not update the resources with the user defined tags. On downgrade: - The status field may remain populated, components may or may not continue From a47189caa6ab4d39e02ea921ac3df26cdf6d9a36 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 8 Sep 2022 17:20:32 +0530 Subject: [PATCH 14/27] updated reviewers --- enhancements/api-review/azure_user_defined_tags.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 8235469b8c..9ce8e58043 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -4,16 +4,15 @@ authors: - "@bhb" reviewers: - "@patrickdillon" ## reviewer for installer component - - "@JoelSpeed" ## reviewer for machine-api-provider-azure component + - "@JoelSpeed" ## reviewer for api and machine-api-provider-azure components - "@dmage" ## reviewer for cluster-image-registry-operator component - "@Miciah" ## reviewer for cluster-ingress-operator component - "@akhil-rane" ## reviewer for cloud-credential-operator component - - TBD ## reviewer for api component - TBD ## reviewer for cloud-network-config-controller component approvers: - "@jerpeter1" ## approver for CFE api-approvers: - - TBD + - "@JoelSpeed" ## approver for api component creation-date: 2022-07-12 last-updated: 2022-07-12 tracking-link: From e19250b307698b656531bdca410e62a3cbd0a759 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Sat, 10 Sep 2022 21:26:09 +0530 Subject: [PATCH 15/27] Updated user defined tags limitation to 10 --- enhancements/api-review/azure_user_defined_tags.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 9ce8e58043..655b916876 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -89,7 +89,7 @@ to meet any of the below conditions through Openshift. - Azure supports a maximum of 50 tags except for Automation, Content Delivery Network, DNS resources which can have a maximum of 15 tags, hence restricting the number of - user defined tags to 5 and 10 for Openshift's internal use, for all the resources + user defined tags to 10 and 5 for Openshift's internal use, for all the resources created by Openshift. All in-cluster operators that create Azure resources (Cluster Infrastructure ,Storage ,Node ,NetworkEdge , Internal Registry ,CCO) will apply these tags during resource creation. @@ -169,10 +169,10 @@ spec: https://docs.microsoft.com/en-us/rest/api/resources/tags for information on tagging Azure resources. Azure supports a maximum of 50 tags per resource except for few, which have limitation of 15 tags. OpenShift - reserves 10 tags for its internal use, and allows 5 tags + reserves 5 tags for its internal use, and allows 10 tags for user configuration. type: array - maxItems: 5 + maxItems: 10 items: description: AzureResourceTag is a tag to apply to Azure resources created for the cluster. type: object From a18de680fafcd0721d546570693b029631307307 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Wed, 14 Sep 2022 12:54:52 +0530 Subject: [PATCH 16/27] Update enhancements/api-review/azure_user_defined_tags.md Co-authored-by: Joel Speed --- enhancements/api-review/azure_user_defined_tags.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 655b916876..d512d07cdd 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -44,9 +44,7 @@ Motivations include but are not limited to: ### User Stories - As an openshift administrator, I want to have tags added to all resources created - in Azure by Openshift. -- As an openshift administrator, I want to restrict access granted to Openshift - specific account. + in Azure by Openshift, so that I can restrict access granted to an OpenShift specific account ### Goals From 37362e6c95db64f15afe9806660ae088f0e90b96 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Wed, 28 Sep 2022 12:23:07 +0530 Subject: [PATCH 17/27] Update azure_user_defined_tags.md --- enhancements/api-review/azure_user_defined_tags.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index d512d07cdd..5237acf63b 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -8,7 +8,7 @@ reviewers: - "@dmage" ## reviewer for cluster-image-registry-operator component - "@Miciah" ## reviewer for cluster-ingress-operator component - "@akhil-rane" ## reviewer for cloud-credential-operator component - - TBD ## reviewer for cloud-network-config-controller component + - "@trozet" ## reviewer for cloud-network-config-controller component approvers: - "@jerpeter1" ## approver for CFE api-approvers: @@ -239,8 +239,8 @@ resource "azurerm_resource_group" "main" { ``` #### Caveats -1. User updates or removes Azure resource tags added by Openshift, using an external interface - will not be reconciled by the resource owning operator. +1. Updating or removing resource tags added by Openshift using an external interface, + may or may not be reconciled by the operator managing the resource. 2. Updating tags of individual resources is not supported and any tag present in `.status.platformStatus.azure.resourceTags` of `infrastructure.config.openshift.io/v1` resource will result in adding tags to all Openshift managed Azure resources. From e924393838ae5e3869efea6ec12e05dba28d6bb0 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 15 Dec 2022 11:35:33 +0530 Subject: [PATCH 18/27] Update azure_user_defined_tags.md --- enhancements/api-review/azure_user_defined_tags.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 5237acf63b..a28842e554 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -44,7 +44,7 @@ Motivations include but are not limited to: ### User Stories - As an openshift administrator, I want to have tags added to all resources created - in Azure by Openshift, so that I can restrict access granted to an OpenShift specific account + in Azure by Openshift, so that I can restrict access granted to an OpenShift specific account. ### Goals @@ -283,8 +283,13 @@ resource "azurerm_resource_group" "main" { ### Graduation Criteria #### Dev Preview -> Tech Preview +- Feature available for end-to-end usage. +- Complete end user documentation. +- UTs and e2e tests are present. +- Gather feedback from the users. #### Tech Preview -> GA +N/A. This feature is for Tech Preview, until decided for GA. #### Removing a deprecated feature From 5cd152284b32fb3ca953b55b031a7c52c9c9e078 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Fri, 16 Dec 2022 10:38:29 +0530 Subject: [PATCH 19/27] Update azure_user_defined_tags.md --- enhancements/api-review/azure_user_defined_tags.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index a28842e554..812e272f27 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -61,7 +61,7 @@ Motivations include but are not limited to: ## Proposal -A tag of the form `kubernetes.io/cluster/:owned` will be added to every +A tag of the form `kubernetes.io_cluster.:owned` will be added to every resource created by Openshift to enable administrator to differentiate the resources created for Openshift cluster. An administrator is not allowed to add or modify the tag having the prefix `kubernetes.io` or `openshift.io` in the name. The same tag can be @@ -238,6 +238,11 @@ resource "azurerm_resource_group" "main" { } ``` +The tag of the form `kubernetes.io_cluster.:owned` added by openshift, where +cluster_id is a string generated by concatenating user inputted cluster name and a random +string will be limited to a maximum length of 27 characters by trimming long cluster name +to 21 characters. + #### Caveats 1. Updating or removing resource tags added by Openshift using an external interface, may or may not be reconciled by the operator managing the resource. From 4ba638f3f7f625445267e79e8e674091243ce7bc Mon Sep 17 00:00:00 2001 From: Bharath B Date: Tue, 20 Dec 2022 11:43:18 +0530 Subject: [PATCH 20/27] Update azure_user_defined_tags.md --- enhancements/api-review/azure_user_defined_tags.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 812e272f27..d79a4ec2c5 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -83,7 +83,7 @@ to meet any of the below conditions special/unicode characters or starting with number, hence these are added as constraints on every other Azure resource required by Openshift as well. 3. A tag value can have a maximum of 256 characters. -4. A resource, resource-group or subscription, user can configure a maximum of 5 tags +4. A resource, resource-group or subscription, user can configure a maximum of 10 tags through Openshift. - Azure supports a maximum of 50 tags except for Automation, Content Delivery Network, DNS resources which can have a maximum of 15 tags, hence restricting the number of From 3d8e2c872554bbc1809f7f8fe79206304d5f2a39 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Mon, 9 Jan 2023 11:21:02 +0530 Subject: [PATCH 21/27] Updated reviewers list --- enhancements/api-review/azure_user_defined_tags.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index d79a4ec2c5..87d9e9e578 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -5,10 +5,8 @@ authors: reviewers: - "@patrickdillon" ## reviewer for installer component - "@JoelSpeed" ## reviewer for api and machine-api-provider-azure components - - "@dmage" ## reviewer for cluster-image-registry-operator component + - "@flavianmissi" ## reviewer for cluster-image-registry-operator component - "@Miciah" ## reviewer for cluster-ingress-operator component - - "@akhil-rane" ## reviewer for cloud-credential-operator component - - "@trozet" ## reviewer for cloud-network-config-controller component approvers: - "@jerpeter1" ## approver for CFE api-approvers: @@ -90,7 +88,7 @@ to meet any of the below conditions user defined tags to 10 and 5 for Openshift's internal use, for all the resources created by Openshift. -All in-cluster operators that create Azure resources (Cluster Infrastructure ,Storage ,Node ,NetworkEdge , Internal Registry ,CCO) will apply these tags during resource creation. +All in-cluster operators that create Azure resources (Cluster Infrastructure ,Storage ,Node ,NetworkEdge , Internal Registry) will apply these tags during resource creation. The userTags field is intended to be set at install time and is considered immutable. Components that respect this field must only ever add tags that they retrieve from this @@ -210,10 +208,8 @@ resources. | Operator | Resources created by the operator | | -------- | ----------------------------- | -| cloud-network-config-controller | Private IP address | | cluster-image-registry-operator | Storage Account | | cluster-ingress-operator | Load Balancer, DNS records | -| cloud-credential-operator | IAM roles and policies | | machine-api-provider-azure | Application Security Group, Availability Set, Group, Load Balancer, Public IP Address, Route, Network Security Group, Virtual Machine Extension, Virtual Interface, Virtual Machine, Virtual Network. | Below list of terraform Azure APIs to create resources should be updated to add user From 5760f0d0353ab1eb0f13df456d434dc23246a981 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Mon, 9 Jan 2023 11:23:24 +0530 Subject: [PATCH 22/27] Update azure_user_defined_tags.md --- enhancements/api-review/azure_user_defined_tags.md | 1 - 1 file changed, 1 deletion(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 87d9e9e578..4a5d452e56 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -257,7 +257,6 @@ to 21 characters. 2. Storage account 3. NetworkEdge 4. Internal Registry - 5. CCO - OpenShift is bound to have the common limitation for all Azure resources created by it and constraints other resources with the least matching limit as below From dd7ce927feff82426bacaa953885b00a19b2d5bc Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 12 Jan 2023 11:41:51 +0530 Subject: [PATCH 23/27] Apply suggestions from code review by Miciah Co-authored-by: Miciah Dashiel Butler Masters --- .../api-review/azure_user_defined_tags.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 4a5d452e56..d9e0eabe0d 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -51,7 +51,7 @@ Motivations include but are not limited to: install config, which is referred and applied by the installer and the in-cluster operators on the the Azure resources during cluster creation. - Tags must be applied at creation time, in an atomic operation. It isn't acceptable - to create an object and to apply tags post cluster creation. + to create an object and to apply tags post resource creation. ### Non-Goals @@ -68,8 +68,8 @@ found applied to other cloud platform resources which supports tagging for ex: A New `userTags` field will be added to `platform.azure` of install-config for the user to define the tags to be added to the resources created by installer and in-cluster operators. -If `platform.azure.userTags` of install-config has any tag defined same will be added -to all the azure resources created by Openshift except, when the tag validation fail +If `platform.azure.userTags` in the install-config has any tags defined, then these tags will be added +to all the azure resources created by OpenShift, except when the tag validation fails to meet any of the below conditions 1. A tag name can have a maximum of 128 characters. - Tag name has a limit of 512 characters for all resources except for @@ -96,9 +96,9 @@ field to cloud resources, they must never remove tags from the existing underlyi resource even if the tags are removed from this field(despite it being immutable). If the userTags field is changed post-install, there is no guarantee about how an -in-cluster operator will respond to the change. Some operators may reconcile the -change and change tags on the Azure resource. Some operators may ignore the change. -However, if tags are removed from userTags, the tag will not be removed from the +in-cluster operator will respond to the change. Operators may reconcile the +change by adding or updating tags on the Azure resource. Operators may ignore the change. +However, if a tag is removed from `userTags`, the tag must not be removed from the Azure resource. ### Workflow Description @@ -108,8 +108,8 @@ Azure resource. - openshift installer validates the tags defined in `.platform.azure.userTags` and adds these tags to all resources created during installation and also updates `.status.platformStatus.azure.resourceTags` of the `infrastructure.config.openshift.io` -- In cluster operators refers `.status.platformStatus.azure.resourceTags` of the - `infrastructure.config.openshift.io` to add tags to resources created later. +- Cluster operators refer to `.status.platformStatus.azure.resourceTags` of the + `infrastructure.config.openshift.io` object to add tags to resources created later. #### Variation [optional] @@ -134,7 +134,7 @@ spec: userTags: additionalProperties: type: string - description: UserTags additional keys and values that the installer + description: UserTags has additional keys and values that the installer will add as tags to all resources that it creates. Resources created by the cluster itself may not include these tags. type: object @@ -218,7 +218,7 @@ defined tags and as well the openshift default tag in the installer component. API update example: A local variable should be defined, which merges the default tag and the user -defined Azure tags, which should be referred in the Azure resource APIs. +defined Azure tags, which should be referenced in the Azure resource APIs. ``` terraform locals { tags = merge( @@ -258,8 +258,8 @@ to 21 characters. 3. NetworkEdge 4. Internal Registry -- OpenShift is bound to have the common limitation for all Azure resources created - by it and constraints other resources with the least matching limit as below +- Because tags in `userTags` must be applied to all Azure resources that OpenShift creates, + tags must fit all the constraints of the various Azure resources as described below: 1. Tag names cannot have `microsoft`, `azure`, `windows` prefixes which are reserved for Azure use. 2. An Azure storage account has a limit of 128 characters for the tag name. @@ -296,9 +296,9 @@ N/A. This feature is for Tech Preview, until decided for GA. ### Upgrade / Downgrade Strategy On upgrade: -- Cluster operators that update the tags of Azure resources created for cluster - should refer the new fields and take action. Any new resource created post-upgrade - and the operators managing the resource will add the user defined tags to the +- Cluster operators that update the tags of Azure resources created for the cluster + should refer to the new fields and take action. For any new resource created post-upgrade, + the operator managing the resource will add the user-defined tags to the resource. But the same does not apply to already existing resources, components may or may not update the resources with the user defined tags. From b1bc2d17983f804402f6d982dff75b313fc14713 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 12 Jan 2023 12:07:58 +0530 Subject: [PATCH 24/27] incorporated comments from Miciah --- .../api-review/azure_user_defined_tags.md | 29 +++++++++++++------ 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index d9e0eabe0d..1f50920983 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -209,7 +209,7 @@ resources. | Operator | Resources created by the operator | | -------- | ----------------------------- | | cluster-image-registry-operator | Storage Account | -| cluster-ingress-operator | Load Balancer, DNS records | +| cluster-ingress-operator | DNS records | | machine-api-provider-azure | Application Security Group, Availability Set, Group, Load Balancer, Public IP Address, Route, Network Security Group, Virtual Machine Extension, Virtual Interface, Virtual Machine, Virtual Network. | Below list of terraform Azure APIs to create resources should be updated to add user @@ -240,7 +240,7 @@ string will be limited to a maximum length of 27 characters by trimming long clu to 21 characters. #### Caveats -1. Updating or removing resource tags added by Openshift using an external interface, +1. Updating or removing resource tags added by Openshift, using an external interface may or may not be reconciled by the operator managing the resource. 2. Updating tags of individual resources is not supported and any tag present in `.status.platformStatus.azure.resourceTags` of `infrastructure.config.openshift.io/v1` resource @@ -251,12 +251,23 @@ to 21 characters. ### Drawbacks - User-defined tags cannot be updated on an Azure resource which is not managed by an operator. In this proposal, the changes proposed and developed will be part of - openshift-* namespace. External operators are not in scope. + openshift-* namespace. External operators are not in the scope. User-defined tags can be updated on the following Azure resources. - 1. Virtual machine resources created for master and worker nodes. - 2. Storage account - 3. NetworkEdge - 4. Internal Registry + - ResouceGroup + - Storage Account + - DNS Zones + - DNS Records + - ApplicationSecurityGroup + - AvailabilitySet + - Group + - LoadBalancer + - PublicIPAddress + - RouteTable + - SecurityGroup + - VirtualMachineExtension + - NetworkInterface + - VirtualMachine + - VirtualNetwork - Because tags in `userTags` must be applied to all Azure resources that OpenShift creates, tags must fit all the constraints of the various Azure resources as described below: @@ -304,8 +315,8 @@ On upgrade: On downgrade: - The status field may remain populated, components may or may not continue - to tag newly created resources with the additional tags depending on regardless of - whether given component still has logic to respect the status tags, after the downgrade. + to tag newly created resources with the additional tags depending on whether + given component still has logic to respect the status tags, after the downgrade. ### Version Skew Strategy From 6d2dbf5d6979bd7d00d92c63e9b8a1e4e76e75d0 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Thu, 12 Jan 2023 13:10:20 +0530 Subject: [PATCH 25/27] fixed lint issues --- .../api-review/azure_user_defined_tags.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index 1f50920983..c2ba778088 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -253,21 +253,21 @@ to 21 characters. operator. In this proposal, the changes proposed and developed will be part of openshift-* namespace. External operators are not in the scope. User-defined tags can be updated on the following Azure resources. - - ResouceGroup - - Storage Account - - DNS Zones - - DNS Records - - ApplicationSecurityGroup - - AvailabilitySet - - Group - - LoadBalancer - - PublicIPAddress - - RouteTable - - SecurityGroup - - VirtualMachineExtension - - NetworkInterface - - VirtualMachine - - VirtualNetwork + - ResouceGroup + - Storage Account + - DNS Zones + - DNS Records + - ApplicationSecurityGroup + - AvailabilitySet + - Group + - LoadBalancer + - PublicIPAddress + - RouteTable + - SecurityGroup + - VirtualMachineExtension + - NetworkInterface + - VirtualMachine + - VirtualNetwork - Because tags in `userTags` must be applied to all Azure resources that OpenShift creates, tags must fit all the constraints of the various Azure resources as described below: From 393b90ebb2e3ae585c5d764860aa372d4f10e3e2 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Wed, 18 Jan 2023 20:52:00 +0530 Subject: [PATCH 26/27] incorporated comments --- .../api-review/azure_user_defined_tags.md | 34 ++++++++++++++----- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index c2ba778088..c6021012a8 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -47,7 +47,7 @@ Motivations include but are not limited to: ### Goals - The administrator or service (in the case of Managed OpenShift) installing OpenShift - can configure allowed number of user-defined tags in the Openshift installer generated + can configure a list of up to 10 user-defined tags in the OpenShift installer generated install config, which is referred and applied by the installer and the in-cluster operators on the the Azure resources during cluster creation. - Tags must be applied at creation time, in an atomic operation. It isn't acceptable @@ -253,7 +253,7 @@ to 21 characters. operator. In this proposal, the changes proposed and developed will be part of openshift-* namespace. External operators are not in the scope. User-defined tags can be updated on the following Azure resources. - - ResouceGroup + - ResourceGroup - Storage Account - DNS Zones - DNS Records @@ -290,9 +290,12 @@ to 21 characters. ### Open Questions ### Test Plan +- Upgrade/downgrade testing +- Sufficient time for feedback +- Available by default +- Stress testing for scaling and tag update scenarios ### Graduation Criteria - #### Dev Preview -> Tech Preview - Feature available for end-to-end usage. - Complete end user documentation. @@ -307,14 +310,27 @@ N/A. This feature is for Tech Preview, until decided for GA. ### Upgrade / Downgrade Strategy On upgrade: -- Cluster operators that update the tags of Azure resources created for the cluster - should refer to the new fields and take action. For any new resource created post-upgrade, - the operator managing the resource will add the user-defined tags to the - resource. But the same does not apply to already existing resources, components may - or may not update the resources with the user defined tags. +- Scenario 1: Upgrade to OpenShift version having support for adding tags. + The new status field won't be populated since it is only populated by the + installer and that can't have happened if the cluster was installed from a + prior version. Components that consume the new field should take no action + since there won't be any tags to apply. +- Scenario 2: Upgrade from OpenShift version having support for adding tags to higher: + Cluster operators that add tags to Azure resources created for the cluster + should refer to the tag fields and take action. For any new resource created + post-upgrade, the operator managing the resource will add the user-defined tags + to the resource. But the same does not apply to already existing resources, + components may or may not update the resources with the user defined tags. On downgrade: -- The status field may remain populated, components may or may not continue +- Scenario 1: Cluster installed with OpenShift version having support for adding tags + and later downgrading to a lower version not having support for addding tags. + The new status field won't be populated since it is only populated by the + installer and that can't have happened if the cluster was installed from a + earlier version and upgraded to version having support for tags. +- Scenario 2: Downgrading a cluster to installed OpenShift version not having support + for adding tags from an OpenShift version having support for addding tags. + The status field may remain populated, components may or may not continue to tag newly created resources with the additional tags depending on whether given component still has logic to respect the status tags, after the downgrade. From 47f5b58a740b0f8a16c98baf5af899b7555c59a4 Mon Sep 17 00:00:00 2001 From: Bharath B Date: Fri, 20 Jan 2023 14:46:35 +0530 Subject: [PATCH 27/27] incorporated comments on downgrade --- .../api-review/azure_user_defined_tags.md | 25 ++++++++++--------- 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/enhancements/api-review/azure_user_defined_tags.md b/enhancements/api-review/azure_user_defined_tags.md index c6021012a8..6ae2a4d98f 100644 --- a/enhancements/api-review/azure_user_defined_tags.md +++ b/enhancements/api-review/azure_user_defined_tags.md @@ -12,7 +12,7 @@ approvers: api-approvers: - "@JoelSpeed" ## approver for api component creation-date: 2022-07-12 -last-updated: 2022-07-12 +last-updated: 2023-01-20 tracking-link: - https://issues.redhat.com/browse/OCPPLAN-8155 - https://issues.redhat.com/browse/CORS-2249 @@ -68,9 +68,9 @@ found applied to other cloud platform resources which supports tagging for ex: A New `userTags` field will be added to `platform.azure` of install-config for the user to define the tags to be added to the resources created by installer and in-cluster operators. -If `platform.azure.userTags` in the install-config has any tags defined, then these tags will be added -to all the azure resources created by OpenShift, except when the tag validation fails -to meet any of the below conditions +If `platform.azure.userTags` in the install-config has any tags defined, then these tags +will be added to all the azure resources created by OpenShift, provided all the defined +tags meet all the below conditions or else the cluster creation will fail. 1. A tag name can have a maximum of 128 characters. - Tag name has a limit of 512 characters for all resources except for storage accounts, which has a limit of 128 characters and hence tag name @@ -323,16 +323,17 @@ On upgrade: components may or may not update the resources with the user defined tags. On downgrade: -- Scenario 1: Cluster installed with OpenShift version having support for adding tags - and later downgrading to a lower version not having support for addding tags. +- Scenario 1: Cluster installed with OpenShift version not having support for adding + tags, upgraded to a version supporting tags and later downgraded to installed version. The new status field won't be populated since it is only populated by the installer and that can't have happened if the cluster was installed from a - earlier version and upgraded to version having support for tags. -- Scenario 2: Downgrading a cluster to installed OpenShift version not having support - for adding tags from an OpenShift version having support for addding tags. - The status field may remain populated, components may or may not continue - to tag newly created resources with the additional tags depending on whether - given component still has logic to respect the status tags, after the downgrade. + earlier version and upgraded to version having support for tags and downgrade will + have no impact with the tags functionality too. +- Scenario 2: Cluster installed with OpenShift version having support for adding tags, + upgraded to higher version and later downgraded to a lower version supporting tags. + The status field may remain populated, components may or may not continue to tag + newly created resources with the additional tags depending on whether given component + sill has the logic to add tags post downgrade. ### Version Skew Strategy