[OSDOCS-19926]: Private hosted clusters on Azure #113083
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| //Module included in the following assemblies: | ||
| // hosted_control_planes/hcp-deploy/hcp-deploy-azure.adoc | ||
|
|
||
| :_mod-docs-content-type: PROCEDURE | ||
| [id="hcp-azure-delete-private_{context}"] | ||
| = Deleting a private hosted cluster on {azure-short} | ||
|
|
||
| [role="_abstract"] | ||
| If you are no longer using a private hosted cluster on {azure-short}, you can delete it. | ||
|
|
||
| The deletion process automatically cleans up Private Link resources in the following order: | ||
|
|
||
| . The Control Plane Operator removes the private endpoint, private DNS zones, VNet links, and A records. | ||
| . The `hcp destroy` command removes role-based access control (RBAC) role assignments. | ||
| . The HyperShift Operator removes Private Link. | ||
|
Member
There was a problem hiding this comment. [non-blocking] The cleanup order is slightly simplified. The
|
||
|
|
||
| .Procedure | ||
|
|
||
| * To delete a private hosted cluster, enter the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ hcp destroy cluster azure \ | ||
| --name ${CLUSTER_NAME} \ | ||
| --azure-creds ${AZURE_CREDS} \ | ||
| --resource-group-name ${MANAGED_RG_NAME} \ | ||
| --dns-zone-rg-name ${DNS_ZONE_RG_NAME} | ||
| ---- | ||
| + | ||
| ** `--name` specifies your hosted cluster name. | ||
| ** `--azure-creds` specifies an {azure-short} credentials file that has permission to create infrastructure resources, such as virtual networks, subnets, and load balancers. This flag is required because the `hcp destroy cluster azure` command cleans up RBAC role assignments before it deletes infrastructure. | ||
| ** `--resource-group-name` specifies the name of the resource group where you created identities. | ||
| ** `--dns-zone-rg-name` specifies the name of the resource group that contains your DNS zone. | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| //Module included in the following assemblies: | ||
| // hosted_control_planes/hcp-deploy/hcp-deploy-azure.adoc | ||
|
|
||
| :_mod-docs-content-type: PROCEDURE | ||
| [id="hcp-azure-private-access_{context}"] | ||
| = Accessing a private {azure-short} hosted cluster | ||
|
|
||
| [role="_abstract"] | ||
| After you create a private hosted cluster, you need to take additional steps to access it. | ||
|
|
||
| .Prerequisites | ||
|
|
||
| * You completed the steps in "Creating a private {azure-short} hosted cluster". | ||
|
|
||
| .Procedure | ||
|
|
||
| * To access a private hosted cluster by generating a `kubeconfig` file, enter the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ hcp create kubeconfig \ | ||
| --name ${CLUSTER_NAME} \ | ||
| --port-forward > ${CLUSTER_NAME}-kubeconfig | ||
| ---- | ||
|
|
||
| * If you have access to the management cluster, you can port forward to the API server to access the private hosted cluster. | ||
| + | ||
| .. Port forward to the `kube-apiserver` service by entering the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ kubectl port-forward svc/kube-apiserver \ | ||
| -n clusters-${CLUSTER_NAME} 6443:6443 & | ||
| ---- | ||
| + | ||
| .. Access the hosted cluster by using the `kubeconfig` file: | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| $ KUBECONFIG=${CLUSTER_NAME}-kubeconfig oc get nodes | ||
| ---- | ||
|
|
||
| * If you have a virtual machine (VM) in an {azure-short} Virtual Network (VNet) that is peered with the hosted cluster's VNet, you can access the API server, but you must first link the private DNS zones to the peered VNet. | ||
| + | ||
| [NOTE] | ||
| ==== | ||
| The Control Plane Operator links private DNS zones only to the hosted cluster's VNet. If you want to resolve the API server hostname from a peered VNet, you must manually link the private DNS zones to that VNet as shown in the following steps. Otherwise, DNS resolution fails from the peered VNet. | ||
| ==== | ||
| + | ||
| .. Link the private DNS zone to your peered VNet as shown in the following example: | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| $ PEERED_VNET_ID="/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/virtualNetworks/<vnet>" | ||
| ---- | ||
| + | ||
| .. Enter the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ az network private-dns link vnet create \ | ||
| --resource-group "${MANAGED_RG_NAME}" \ | ||
| --zone-name "${CLUSTER_NAME}.hypershift.local" \ | ||
| --name "peered-vnet-link" \ | ||
| --virtual-network "${PEERED_VNET_ID}" \ | ||
| --registration-enabled false | ||
| ---- | ||
| + | ||
| .. If you also need base-domain resolution, enter the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ az network private-dns link vnet create \ | ||
| --resource-group "${MANAGED_RG_NAME}" \ | ||
| --zone-name "${PARENT_DNS_ZONE}" \ | ||
| --name "peered-vnet-basedomain-link" \ | ||
| --virtual-network "${PEERED_VNET_ID}" \ | ||
| --registration-enabled false | ||
| ---- | ||
| + | ||
| .. Access the cluster as shown in the following example: | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| $ KUBECONFIG=${CLUSTER_NAME}-kubeconfig oc get nodes | ||
| ---- |
|
lahinson marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| //Module included in the following assemblies: | ||
| // hosted_control_planes/hcp-deploy/hcp-deploy-azure.adoc | ||
|
|
||
| :_mod-docs-content-type: PROCEDURE | ||
| [id="hcp-azure-private-hosted_{context}"] | ||
| = Creating a private {azure-short} hosted cluster | ||
|
|
||
| [role="_abstract"] | ||
| Create a private hosted cluster to ensure that the communication between compute nodes and the hosted control plane occurs over {azure-short} Private Link. | ||
|
|
||
| .Prerequisites | ||
|
Contributor
There was a problem hiding this comment. Since this procedure seems to rely on all of the environment variables from the previous steps, you might want to add an explicit prerequisite stating this. If a user completes the previous steps in a separate terminal, or they completed the steps on a prior day and then closed their terminal, the environment variables would be lost.
Contributor
Author
There was a problem hiding this comment. Good point. I'll mention the environment variables in the prereqs. |
||
|
|
||
| * You configured a subnet, as described in "Preparing a subnet for a private hosted cluster on {azure-short}". | ||
|
|
||
| * You installed the HyperShift Operator on an {product-title} management cluster on {azure-short}, as described in "Installing the HyperShift Operator with private platform support". | ||
|
|
||
| * You created Identity and Access Management (IAM) resources, as described in "Configuring IAM resources for a private hosted cluster". | ||
|
|
||
| * You created infrastructure, as described in "Creating infrastructure for a private hosted cluster". | ||
|
|
||
| * This procedure relies on environment variables that you created in the prerequisite procedures. Be sure to complete this procedure in the same terminal where you already defined the environment variables. | ||
|
|
||
| .Procedure | ||
|
|
||
| . Create the private hosted cluster by entering the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ hcp create cluster azure \ | ||
| --name "$CLUSTER_NAME" \ | ||
| --namespace "clusters" \ | ||
| --azure-creds ${AZURE_CREDS} \ | ||
| --location ${LOCATION} \ | ||
| --node-pool-replicas 2 \ | ||
| --base-domain ${PARENT_DNS_ZONE} \ | ||
| --pull-secret ${PULL_SECRET} \ | ||
| --generate-ssh \ | ||
| --release-image ${RELEASE_IMAGE} \ | ||
| --resource-group-name "${MANAGED_RG_NAME}" \ | ||
| --vnet-id "${VNET_ID}" \ | ||
| --subnet-id "${SUBNET_ID}" \ | ||
| --network-security-group-id "${NSG_ID}" \ | ||
| --sa-token-issuer-private-key-path "${SA_TOKEN_ISSUER_PRIVATE_KEY_PATH}" \ | ||
| --oidc-issuer-url "${OIDC_ISSUER_URL}" \ | ||
| --dns-zone-rg-name ${DNS_ZONE_RG_NAME} \ | ||
| --assign-service-principal-roles \ | ||
| --workload-identities-file ${WORKLOAD_IDENTITIES_FILE} \ | ||
| --external-dns-domain ${DNS_ZONE_NAME} \ | ||
| --endpoint-access Private \ | ||
|
There was a problem hiding this comment. When choosing the For reference:
As far as I know, the issue is still present. It might be worth adding a note to help users avoid this condition when picking their DNS zone name. @bryan-cox any additional context on the current state?
Contributor
Author
There was a problem hiding this comment. I added a note below the command. |
||
| --endpoint-access-private-nat-subnet-id "${NAT_SUBNET_ID}" | ||
| ---- | ||
| + | ||
| * When you choose a value for the `--external-dns-domain` flag, ensure that the value does not match `{cluster-name}.{base-domain}`. If you use `{cluster-name}.{base-domain}` for the value, the Control Plane Operator creates a private DNS zone that can shadow the `*.apps` domain, which causes the console and ingress to become unreachable. | ||
| * `--endpoint-access` accepts 3 values: | ||
| + | ||
| ** `Public`, which is the default value. When you specify `Public`, the API server is accessible through public endpoint only. | ||
| ** `PublicAndPrivate`, where the API server is accessible through both public and private endpoints. | ||
| ** `Private`, where the API server is accessible only through Private Link. | ||
| + | ||
| After you create a cluster, you cannot change it between `Public` endpoint and non-public (`PublicAndPrivate` or `Private`) endpoint access. | ||
| + | ||
| If you need to allow `Private` endpoint connections from {azure-short} subscriptions other than the hosted cluster's subscription, use the following flag: `--endpoint-access-private-additional-allowed-subscriptions "sub-id-1, sub-id-2"`. | ||
|
|
||
| .Verification | ||
|
|
||
| . Check the Private Link resources by entering the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ oc get azureprivatelinkservices -n clusters-${CLUSTER_NAME} | ||
| ---- | ||
|
|
||
| . Check the status and connections by entering the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ oc get azureprivatelinkservices -n clusters-${CLUSTER_NAME} -o yaml | ||
| ---- | ||
| + | ||
| .Setup progress conditions | ||
| [cols="2",options="header"] | ||
| |=== | ||
| |Condition |Description | ||
|
|
||
| |`AzureInternalLoadBalancerAvailable` | ||
| |The internal load balancer has a front-end IP address. | ||
|
|
||
| |`AzurePLSCreated` | ||
| |Private Link is created in the management cluster. | ||
|
|
||
| |`AzurePrivateEndpointAvailable` | ||
| |Private endpoint is created in the hosted cluster VNet. | ||
|
|
||
| |`AzurePrivateDNSAvailable` | ||
| |Private DNS zones and A records are created. | ||
|
|
||
| |`AzurePrivateLinkServiceAvailable` | ||
| |All components are ready and private connectivity is available. | ||
|
|
||
| |=== | ||
|
|
||
| . Check the overall cluster status by entering the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ oc get hostedcluster ${CLUSTER_NAME} -n clusters | ||
| ---- | ||
|
|
||
| . Wait for the status by entering the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ oc wait --for=condition=Available hostedcluster/${CLUSTER_NAME} -n clusters --timeout=30m | ||
| ---- | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[suggestion] The deletion flow described is accurate for Private Link resource cleanup, but it's worth noting that
hcp destroy cluster azurealso cleans up RBAC role assignments before deleting infrastructure. This is why the--azure-credsflag is required for the destroy command.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a statement about that.