Implement ccoctl command to create infrastructure required for Azure workload identity

[abutcher]

Implement ccoctl azure sub-commands which will create an RSA key pair, OIDC Azure blob container infrastructure and user-assigned managed identities for processed CredentialsRequests which must contain .spec.serviceAccountNames to receive federated identity credentials for the listed service accounts.

➜  ccoctl azure -h
Creating/updating/deleting cloud credentials objects for Azure

Usage:
  ccoctl azure [command]

Available Commands:
  create-all                Create OIDC issuer and managed identities
  create-key-pair           Create a key pair
  create-managed-identities Create Azure Managed Identities
  create-oidc-issuer        Create OIDC Issuer
  delete                    Delete OIDC issuer and managed identities

Flags:
  -h, --help   help for azure

Use "ccoctl azure [command] --help" for more information about a command.

ccoctl azure create-key-pair

➜  ccoctl azure create-key-pair --output-dir /tmp/ccoctl-output

ccoctl azure create-oidc-issuer

OIDC issuer infrastructure will be created in a resource group with a name derived from --name when no --oidc-resource-group-name is provided. By default this OIDC resource group will be named <--name parameter> + "-oidc", eg "abutchertest-oidc" but may be explicitly named by providing an --oidc-resource-group-name parameter.

➜  ccoctl azure create-oidc-issuer \
        --name abutchertest \
        --output-dir /tmp/ccoctl-output \
        --region centralus \
        --subscription-id <subscription-id> \
        --public-key-file /tmp/ccoctl-output/serviceaccount-signer.public
        [--oidc-resource-group-name <oidc rg name>]
        [--storage-account-name <storage account name>]
        [--user-tags key1=value1,key2=value2]

Note: The storage account name has more strict requirements than that of a resource group name. For example, if the storage account name derived from the --name parameter is invalid the command will exit earlier with an error. If specific naming is required --storage-account-name can be specified explicitly.

➜  ccoctl azure create-oidc-issuer --name abutcher-demo \
          --output-dir /tmp/ccoctl-output \
          --region centralus \
          --subscription-id <subscription-id> \
          --public-key-file /tmp/ccoctl-output/serviceaccount-signer.public \
          --user-tags cheese=beans,corn=rice
2023/05/31 10:40:37 No --oidc-resource-group-name provided, defaulting OIDC resource group name to abutcher-demo-oidc
2023/05/31 10:40:37 No --storage-account-name provided, defaulting storage account name to abutcher-demo
2023/05/31 10:40:37 invalid storage account name: abutcher-demo. Azure storage account names must be between 3 and 24 characters in length and may contain numbers and lowercase letters only.

ccoctl azure create-managed-identities

ccoctl will grant permissions to created user-assigned managed identities within the scope of an "installation" resource group and this resource group must be used as the resource group configured for future cluster installation. By default this installation resource group will be named <--name parameter>, eg "abutchertest" but may be explicitly named by providing an --installation-resource-group-name parameter.

Note: The OpenShift installer requires that the installation resource group be entirely empty so ccoctl just creates the resource group such that the resource group can be used for scoping user-assigned managed identities and instructs that this resource group MUST be used for cluster installation. Being able to provide the installation resource group for scoping allows us to use ccoctl to create OIDC/managed identity infrastructure for an existing cluster to assist with testing.

In order to scope cluster ingress operations, ccoctl must also be provided the --dnszone-resource-group-name which is the name of the resource group in which the future cluster's base domain DNS zone exists (as provided to the OpenShift installer via the install-config.yaml).

➜   ccoctl azure create-managed-identities \
         --name abutchertest \
         --output-dir /tmp/ccoctl-output \
         --region centralus \
         --subscription-id <subscription-id> \
         --credentials-requests-dir /tmp/azure-crs \
         --resource-group abutchertest-infra \
         --issuer-url https://abutchertest.blob.core.windows.net/abutchertest/ \
         --dnszone-resource-group-name <future cluster's base domain resource group name>
         [--oidc-resource-group-name <oidc rg name>]
         [--installation-resource-group-name <installation rg name>]
         [--user-tags key1=value1,key2=value2]

ccoctl azure create-all

ccoctl azure create-all combines create-key-pair, create-oidc-issuer and create-managed-identities into a single command.

➜   ccoctl azure create-all \
         --name abutchertest \
         --output-dir /tmp/ccoctl-output \
         --region centralus \
         --subscription-id <subscription-id> \
         --credentials-requests-dir /tmp/azure-crs \
         --dnszone-resource-group-name <future cluster's base domain resource group name> \
         [--oidc-resource-group-name <oidc rg name>]
         [--installation-resource-group-name <installation rg name>]
         [--storage-account-name <storage account name>]
         [--user-tags key1=value1,key2=value2]

ccoctl azure delete

The delete subcommand deletes the storage account, blob container and managed identities from the OIDC resource group but will not delete the OIDC resource group unless requested. Note that as above, the OIDC resource group name may be specified by --oidc-resource-group-name but is defaulted to <--name parameter> + "-oidc".

➜   ccoctl azure delete \
         --name abutchertest \
         --region centralus \
         --subscription-id <subscription-id>
         [--oidc-resource-group-name <oidc rg name>]
         [---delete-oidc-resource-group]

CCO-232
openshift/enhancements#1301

[codecov]

Codecov Report

Merging #523 (7ba2acc) into master (38e7f96) will decrease coverage by 0.27%.
The diff coverage is 43.09%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #523      +/-   ##
==========================================
- Coverage   48.80%   48.54%   -0.27%     
==========================================
  Files          88       93       +5     
  Lines        9853    11422    +1569     
==========================================
+ Hits         4809     5545     +736     
- Misses       4469     5259     +790     
- Partials      575      618      +43     

Impacted Files	Coverage Δ
pkg/cmd/provisioning/azure/azure.go	0.00% <0.00%> (ø)
pkg/cmd/provisioning/azure/create_all.go	0.00% <0.00%> (ø)
pkg/cmd/provisioning/azure/delete.go	0.00% <0.00%> (ø)
pkg/azure/clients.go	4.93% <5.44%> (+4.93%)	⬆️
pkg/cmd/provisioning/azure/create_oidc_issuer.go	53.21% <53.21%> (ø)
pkg/cmd/provisioning/utils.go	77.77% <57.14%> (+0.32%)	⬆️
...md/provisioning/azure/create_managed_identities.go	57.62% <57.62%> (ø)
pkg/azure/mock/client_generated.go	71.20% <69.00%> (-28.80%)	⬇️

... and 1 file with indirect coverage changes

[abutcher]

UT, create-all and delete sub-commands still on the way but would appreciate comments on what I have so far.

/assign @jstuever
/cc @2uasimojo

[jstuever]

Some thoughts during a partial review...

[RomanBednar]

@abutcher Does this statement still stand? Can we expect to get azure_federated_token_file in the cloud credentials secret or do we have to assume the token path /var/run/secrets/openshift/serviceaccount?

[abutcher]

@RomanBednar We'll set azure_federated_token_file in the secret as /var/run/secrets/openshift/serviceaccount/token.

cloud-credential-operator/pkg/cmd/provisioning/azure/create_managed_identities.go

Line 253 in 565bf56

fileData := fmt.Sprintf(secretManifestTemplate, clientID, tenantID, region, subscriptionID, provisioning.OidcTokenPath, cr.Spec.SecretRef.Name, cr.Spec.SecretRef.Namespace)

cloud-credential-operator/pkg/cmd/provisioning/azure/create_managed_identities.go

Line 42 in 565bf56

azure_federated_token_file: %s

cloud-credential-operator/pkg/cmd/provisioning/constants.go

Line 17 in 565bf56

OidcTokenPath = "/var/run/secrets/openshift/serviceaccount/token"

[abutcher]

/test e2e-upgrade

[jstuever]

/test verify

[abutcher]

/test verify

[jeana-redhat]

/label docs-approved
Great explanations here and in the enhancement, tysm!

[jstuever]

/test e2e-azure-manual-oidc

[jstuever]

/test e2e-azure-manual-oidc

[jstuever]

/test e2e-azure-manual-oidc

[jstuever]

/test e2e-azure-manual-oidc

[jstuever]

/test e2e-azure-manual-oidc

[jstuever]

/test e2e-azure-manual-oidc

[openshift-ci]

@abutcher: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-manual-oidc	7ba2acc	link	false	/test e2e-azure-manual-oidc

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[jstuever]

e2e-azure-manual-oidc did what we expected at this point. It was able to create the oidc resources with ccoctl azure create-all, and begin installing a cluster. It failed because not all of the operators at this point in time have the necessary changes to support oidc. As a result, those operators failed because their client authentication secrets do not contain a clientSecret.

[jstuever]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abutcher, jstuever

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [abutcher,jstuever]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment