diff --git a/docs/user/azure/README.md b/docs/user/azure/README.md index 635b8dd74ce..62a1c6aaefc 100644 --- a/docs/user/azure/README.md +++ b/docs/user/azure/README.md @@ -16,4 +16,8 @@ Please see the [Issue Tracker][issues] for current known issues. Please report a new issue if you do not find an issue related to any trouble you're having. +## Azure MAG + +Please see [Azure MAG Instructions](install_upi_azuremag.md), this is currently in pre-alpha. + [issues]: https://github.com/openshift/installer/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+azure diff --git a/docs/user/azure/install_upi_azuremag.md b/docs/user/azure/install_upi_azuremag.md new file mode 100644 index 00000000000..d509e710951 --- /dev/null +++ b/docs/user/azure/install_upi_azuremag.md @@ -0,0 +1,181 @@ +# Azure MAG UPI Instructions + +This brief guide will demonstrate how to use the UPI installer to install OCP 4.X in Azure Government + +## Notice of supportability + +CloudFit Software, partnering with Microsoft and Red Hat are actively working towards support exceptions that will cover OCP 4.3 implementations running on Azure Government implementations. This information will be published and updated periodically. +Until there is a support excpetion Red Hat recommends that this only be used in proof of concept (POC) environments, and does not recommend this for production use cases as it will not be supported. + +## Install Procedures + +### Prerequisites + +Applications in your $PATH +- Azcli +- jq +- yq +- oc client +- kubectl +- openshift-install + +Azure Access +- Contributor access to an existing azure government subscription +- Ability to create the following resource types + - Disk + - DNS Zone + - Image + - Load Balancer + - Managed Identity + - Network Interface + - With a public ip binding (this will be resolved in a later release) + - Network Security Group + - Private DNS Zone + - Public IP address + - Storage Account + - Virtual Machine + - Virtual Network + +Azure Rquirements +- Public DNS zone available in azure government + +### Create Install Config + +Here is a sample install config, create a install-config.yaml file in an empty folder with this template to get started. + +```yaml +#install-config.yaml +apiVersion: v1 +baseDomain: +compute: +- hyperthreading: Enabled + name: worker + platform: {} + replicas: 3 +controlPlane: + hyperthreading: Enabled + name: master + platform: {} + replicas: 3 +metadata: + creationTimestamp: null + name: +networking: + clusterNetwork: + - cidr: 10.128.0.0/14 + hostPrefix: 23 + machineNetwork: + - cidr: 10.0.0.0/16 + networkType: OpenShiftSDN + serviceNetwork: + - 172.30.0.0/16 +platform: + azure: + baseDomainResourceGroupName: + region: +publish: External +pullSecret: '' +sshKey: | + +``` + +Create or modify this file to ensure the right azure environment secrets get passed, ~/.azure/osServicePrincipal.json. All of these values are in plain text, they will be translated to base64 encoded secrets during the install. + +```json +{ + "subscriptionId":"", + "clientSecret":"", + "tenantId":"" +} +``` + + +### Copy required files + +Copy the following files from $CODE_LOCATION/upi/azure + +- 01_vnet.json +- 02_storage.json +- 03_infra.json +- 04_bootstrap.json +- 05_masters.json +- 06_workers.json +- azureGovQuickstart.sh + +and make the azureGovQuickstart.sh file executable. + +A quick way to do this is (assuming default code location) +```shell +export CODE_LOCATION=~/go/src/github.com/openshift/installer +cp $CODE_LOCATION/upi/azure/0*.json ./ +cp $CODE_LOCATION/upi/azure/azureGovQuickstart.sh ./ +chmod +x azureGovQuickstart.sh +``` + +### Run Azure Gov Quickstart + +Be in the directory with the copied files and run +```shell +export WORKER_NODE_COUNT= +./azureGovQuickstart.sh -w $WORKER_NODE_COUNT +``` + +#### Log into azure portal +The installer will prompt you to sign into the azure gov portal with a web browser pop up. + +#### Finish Cluster Configuration + +Wait 10 minutes for the nodes to requst cluster membership and then run the following. + +After the script completes configure kubeconfig, join the work nodes, edit the registry operator, and launch the web console. + +```bash +# Configure kubeconfig to authenticate against new openshift +export KUBECONFIG="$PWD/auth/kubeconfig" + +# Get CSR ids of pending requests +oc get csr -A + +# Approve CSR ids +oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve + +# Nodes should populate and become ready in a couple of minutes +watch oc get nodes + +# Edit image registry operator config +# The storage operator doesn't yet work with Azure Mag so the internal registry has to be disabled +oc edit configs.imageregistry.operator.openshift.io/cluster +# change managedState: Managed --> Removed +``` + +Add DNS Records in public and private dns zone of the ip address assigned to the new load balancer. + +*.apps --> ip address of new LB + +```bash +# Complete the cluster install and get temporary admin password for web console +openshift-install wait-for install-complete +``` + +#### Add Azure Disk Storage Class + +Add the following storage class to your deployment, for more information see official [docs](https://kubernetes.io/docs/concepts/storage/storage-classes/#azure-disk-storage-class) + +```yaml +# azure-disk.yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + annotations: + description: azure disk + storageclass.kubernetes.io/is-default-class: "true" + name: azuredisk +parameters: + kind: managed + location: + skuName: +provisioner: kubernetes.io/azure-disk +reclaimPolicy: Delete +volumeBindingMode: Immediate +``` \ No newline at end of file diff --git a/docs/user/azure/install_upi_azuremag_quickstart.md b/docs/user/azure/install_upi_azuremag_quickstart.md new file mode 100644 index 00000000000..ed61efe9787 --- /dev/null +++ b/docs/user/azure/install_upi_azuremag_quickstart.md @@ -0,0 +1,31 @@ +# Azure MAG UPI quickstart + +1. run ```openshift-install create install-config``` + - input valid azure public information (this won't be used) +2. Modify install-config with the correct values for azure government + - platform.azure.region + - platform.azure.baseDomainResourceGroupName + - basedomain +3. copy all ARM templates from $CODE_LOCATION/upi/azure +4. copy azureGovQuickstart.sh from $CODE_LOCATION/upi/azure +5. run azureGovQuickstart.sh -w {Number of desired worker nodes} +6. when installer pauses edit the following files + 1. manifests/cloud-provider-config with the correct information + - cloud: AzurePublicCloud --> AzureUSGovernmentCloud + - tenantId: {Azure Public tenant} --> Azure Gov Tenant + - subscriptionId: {Azure Public Subscription} --> Azure Gov Subscription + 2. openshift/99_cloud-creds-secret with the correct information + - azure_subscription_id: {base64 encoded Public subscription} --> base64 Azure Gov Subscription + - azure_client_id: {base64 encoded Client ID} --> base64 Azure Gov client id + - azure_client_secret: {base64 encoded Client Secret} --> base64 Azure Gov Client Secret + - azure_tenant_id: {base64 encoded tenant id} --> base64 Azure Gov tenant id +7. run ```export KUBECONFIG="$PWD/auth/kubeconfig"``` +8. watch for incoming CSRs with ```watch oc get csr -A``` + - will look like this "system:serviceaccount:openshift-machine-config-operator:node-bootstrapper" +9. Approve CSRs with ```oc adm certificate approve [csr IDs]``` +10. after CSRs have been approved you should have 3 master nodes and desired worker nodes ```oc get nodes``` +11. run ```oc edit configs.imageregistry.operator.openshift.io/cluster ``` + - change managedState: Managed --> Removed +12. add A records to the public and private dns zone for the created node LB + - *.apps.{base domain} +13. run ```openshift-install wait-for install-complete``` for the username and password of the web console diff --git a/pkg/asset/installconfig/azure/azure.go b/pkg/asset/installconfig/azure/azure.go index 5f49b212019..8a8f6618d36 100644 --- a/pkg/asset/installconfig/azure/azure.go +++ b/pkg/asset/installconfig/azure/azure.go @@ -16,7 +16,7 @@ import ( ) const ( - defaultRegion string = "eastus" + azureEnvironment string = "AZURE_ENVIRONMENT" ) // Platform collects azure-specific configuration. @@ -47,11 +47,6 @@ func Platform() (*azure.Platform, error) { return strings.SplitN(s, " ", 2)[0] }) - _, ok := regions[defaultRegion] - if !ok { - return nil, errors.Errorf("installer bug: invalid default azure region %q", defaultRegion) - } - sort.Strings(longRegions) sort.Strings(shortRegions) @@ -61,7 +56,6 @@ func Platform() (*azure.Platform, error) { Prompt: &survey.Select{ Message: "Region", Help: "The azure region to be used for installation.", - Default: fmt.Sprintf("%s (%s)", defaultRegion, regions[defaultRegion]), Options: longRegions, }, Validate: survey.ComposeValidators(survey.Required, func(ans interface{}) error { @@ -89,7 +83,8 @@ func getRegions() (map[string]string, error) { if err != nil { return nil, err } - + client := azsub.NewClientWithBaseURI(session.Environment.ResourceManagerEndpoint) + client.Authorizer = session.Authorizer ctx, cancel := context.WithTimeout(context.TODO(), 1*time.Minute) defer cancel() @@ -110,7 +105,8 @@ func getResourceCapableRegions() ([]string, error) { if err != nil { return nil, err } - + client := azres.NewProvidersClientWithBaseURI(session.Environment.ResourceManagerEndpoint, session.Credentials.SubscriptionID) + client.Authorizer = session.Authorizer ctx, cancel := context.WithTimeout(context.TODO(), 1*time.Minute) defer cancel() diff --git a/pkg/asset/installconfig/azure/dns.go b/pkg/asset/installconfig/azure/dns.go index 50552d6e6a8..46255e4ac1d 100644 --- a/pkg/asset/installconfig/azure/dns.go +++ b/pkg/asset/installconfig/azure/dns.go @@ -118,7 +118,7 @@ func NewDNSConfig() (*DNSConfig, error) { } func newZonesClient(session *Session) ZonesGetter { - azureClient := azdns.NewZonesClient(session.Credentials.SubscriptionID) + azureClient := azdns.NewZonesClientWithBaseURI(session.Environment.ResourceManagerEndpoint, session.Credentials.SubscriptionID) azureClient.Authorizer = session.Authorizer return &ZonesClient{azureClient: azureClient} } diff --git a/pkg/asset/installconfig/azure/session.go b/pkg/asset/installconfig/azure/session.go index 4d0ed40b1ce..70ba9083da7 100644 --- a/pkg/asset/installconfig/azure/session.go +++ b/pkg/asset/installconfig/azure/session.go @@ -20,6 +20,7 @@ const azureAuthEnv = "AZURE_AUTH_LOCATION" var ( defaultAuthFilePath = filepath.Join(os.Getenv("HOME"), ".azure", "osServicePrincipal.json") onceLoggers = map[string]*sync.Once{} + defaultAzureEnvironment = azureenv.PublicCloud ) //Session is an object representing session for subscription @@ -27,6 +28,7 @@ type Session struct { GraphAuthorizer autorest.Authorizer Authorizer autorest.Authorizer Credentials Credentials + Environment azureenv.Environment } //Credentials is the data type for credentials as understood by the azure sdk @@ -37,6 +39,21 @@ type Credentials struct { TenantID string `json:"tenantId,omitempty"` } +func getEnvironmentFromOS() azureenv.Environment { + env := defaultAzureEnvironment + if envSet := os.Getenv(azureEnvironment); len(envSet) > 0 { + logrus.Debugf("Found Azure environment override, trying to set environment to %v", envSet) + foundEnvironment, err := azureenv.EnvironmentFromName(envSet) + if err == nil { + env = foundEnvironment + logrus.Debugf("Successfully set azure environment to %v", foundEnvironment.Name) + }else { + logrus.Debugf("Error setting environment, %v", err.Error()) + } + } + return env +} + // GetSession returns an azure session by using credentials found in ~/.azure/osServicePrincipal.json // and, if no creds are found, asks for them and stores them on disk in a config file func GetSession() (*Session, error) { @@ -44,14 +61,15 @@ func GetSession() (*Session, error) { if f := os.Getenv(azureAuthEnv); len(f) > 0 { authFile = f } - return newSessionFromFile(authFile) + env := getEnvironmentFromOS() + return newSessionFromFile(authFile, env) } -func newSessionFromFile(authFilePath string) (*Session, error) { +func newSessionFromFile(authFilePath string, env azureenv.Environment) (*Session, error) { // NewAuthorizerFromFileWithResource uses `auth.GetSettingsFromFile`, which uses the `azureAuthEnv` to fetch the auth credentials. // therefore setting the local env here to authFilePath allows NewAuthorizerFromFileWithResource to load credentials. os.Setenv(azureAuthEnv, authFilePath) - _, err := auth.NewAuthorizerFromFileWithResource(azureenv.PublicCloud.ResourceManagerEndpoint) + _, err := auth.NewAuthorizerFromFileWithResource(env.ResourceManagerEndpoint) if err != nil { logrus.Debug("Could not get an azure authorizer from file. Asking user to provide authentication info") credentials, err := askForCredentials() @@ -82,12 +100,12 @@ func newSessionFromFile(authFilePath string) (*Session, error) { logrus.Infof("Credentials loaded from file %q", authFilePath) }) - authorizer, err := authSettings.ClientCredentialsAuthorizerWithResource(azureenv.PublicCloud.ResourceManagerEndpoint) + authorizer, err := authSettings.ClientCredentialsAuthorizerWithResource(env.ResourceManagerEndpoint) if err != nil { return nil, errors.Wrap(err, "failed to get client credentials authorizer from saved azure auth settings") } - graphAuthorizer, err := authSettings.ClientCredentialsAuthorizerWithResource(azureenv.PublicCloud.GraphEndpoint) + graphAuthorizer, err := authSettings.ClientCredentialsAuthorizerWithResource(env.GraphEndpoint) if err != nil { return nil, errors.Wrap(err, "failed to get GraphEndpoint authorizer from saved azure auth settings") } @@ -96,6 +114,7 @@ func newSessionFromFile(authFilePath string) (*Session, error) { GraphAuthorizer: graphAuthorizer, Authorizer: authorizer, Credentials: *credentials, + Environment: env, }, nil } diff --git a/upi/azure/azureGovQuickstart.sh b/upi/azure/azureGovQuickstart.sh new file mode 100755 index 00000000000..b883cb1a2d6 --- /dev/null +++ b/upi/azure/azureGovQuickstart.sh @@ -0,0 +1,283 @@ +#!/bin/bash + +# Initialize global variables +VERBOSE=no +LOGOUTPUT=no +WORKERNODES=3 +OCPRELEASE=4.3 +LOGFILE=quickstart.log + +function try() +{ + [[ $- = *e* ]]; SAVED_OPT_E=$? + set +e +} + +function throw() +{ + while [[ "$1" != "" ]]; do + echo $1 > /dev/stderr + shift + done + exit 1 +} + +function catch() +{ + export ex_code=$? + (( $SAVED_OPT_E )) && set +e + return $ex_code +} + +function throwErrors() +{ + set -e +} + +function ignoreErrors() +{ + set +e +} + +function cleanupError() +{ + set +x +} + +function help() +{ + echo "Create UPI OCP $OCPRELEASE on Azure MAG, see Readme" + echo " " + echo "options:" + echo "-h, --help show brief help" + echo "-w, --worker-nodes=number specify number of worker nodes, 3 default" + echo "--login Perform az login prompting for user credentials" + echo "--verbose Verbose output from Azure commands" + echo "--log-output Log output to a file" + exit 0 + } + +function echoDo() +# $1: Description +# $2: Command +{ + redirect="" + if [ "$LOGOUTPUT" == "yes" ]; then + redirect="| tee -a ${LOGFILE}" + fi + echo "Task: $1" $redirect + shift + C='' + for i in "$@"; do + i="${i//\\/\\\\}" + C="$C \"${i//\"/\\\"}\"" + done + echo $C + eval $C $redirect + res=$? + echo "--->> Result Code $res" $redirect + return $res +} + +# Display help before checking for executables called during parameter checks +if [ "$1" == "-h" -o "$1" == "--help" ]; then + help + exit 0 +fi + +echo "Making sure all the correct applications are in your path" +echo "checking az cli" +which az &>/dev/null || throw "Az cli is not installed, please install. https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest" +echo "checking jq" +which jq &>/dev/null || throw "jq is not installed, please install. https://stedolan.github.io/jq/download/" +echo "checking yq" +which yq &>/dev/null || throw "jq is not installed, please install. https://github.com/mikefarah/yq/releases" +echo "checking oc client" +which oc &>/dev/null || throw "oc is not installed, please install. https://docs.openshift.com/container-platform/$OCPRELEASE/cli_reference/openshift_cli/getting-started-cli.html#cli-installing-cli_cli-developer-commands" +echo "checking kubectl" +which kubectl &>/dev/null || throw "kubectl is not installed, please install. https://kubernetes.io/docs/tasks/tools/install-kubectl/" +echo "checking openshift-install" +which openshift-install &>/dev/null || throw "openshift-install is not installed, please install. https://docs.openshift.com/container-platform/$OCPRELEASE/installing/installing_azure/installing-azure-default.html#installation-obtaining-installer_installing-azure-default" + +while test $# -gt 0; do + case "$1" in + -h|--help) + help + exit 0 + ;; + -w) + shift + if test $# -gt 0; then + WORKERNODES=$1 + else + WORKERNODES=3 + fi + ;; + --worker-nodes*) + WORKERNODES=$(echo $1 | sed -e 's/^[^=]*=//g') + ;; + --login) + az login || throw "Unable to login to azure" + ;; + --verbose) + VERBOSE=yes + ;; + --log-output) + LOGOUTPUT=yes + ;; + *) + break + ;; + esac + shift +done + +# Begin processing +throwErrors + +az vm list -o none &>> /dev/null +azure_ok=$? +echo "Azure Check result $azure_ok" +if [[ $azure_ok > 0 ]]; then + throw "No valid Azure session. Try az login again" + exit 1 +fi +az cloud set -n AzureUSGovernment + +azout="-o none" + +if [[ "$VERBOSE" == "yes" ]]; then + azout="-o json" +fi + +if [ ! -f install-config.yaml ]; then + echo Missing "install-config.yaml" >/dev/stderr + echo Please create install-config.yaml using template in README.md > /dev/stderr + exit 1 +fi + +CLUSTER_NAME=$(yq -r .metadata.name install-config.yaml) +AZURE_REGION=$(yq -r .platform.azure.region install-config.yaml) +SSH_KEY=$(yq -r .sshKey install-config.yaml | xargs) +BASE_DOMAIN=$(yq -r .baseDomain install-config.yaml) +BASE_DOMAIN_RESOURCE_GROUP=$(yq -r .platform.azure.baseDomainResourceGroupName install-config.yaml) + +if [ "$CLUSTER_NAME" == "" -o "$BASE_DOMAIN" == "example.com" ]; then + echo CLUSTER_NAME: $CLUSTER_NAME + echo BASE_DOMAIN: $BASE_DOMAIN + throw "install-config.yaml is not properly configured. Please fill it out before running openshift-install" + exit 1 +fi + +python3 -c ' +import yaml; +path = "install-config.yaml"; +data = yaml.full_load(open(path)); +data["compute"][0]["replicas"] = 0; +open(path, "w").write(yaml.dump(data, default_flow_style=False))' +openshift-install create manifests || throw "Unable to create manifests" +rm -fv openshift/99_openshift-cluster-api_master-machines-*.yaml +rm -fv openshift/99_openshift-cluster-api_worker-machineset-*.yaml +# Change cloud provider +sed -i 's/AzurePublicCloud/AzureUSGovernmentCloud/' manifests/cloud-provider-config.yaml + +python3 -c ' +import yaml; +path = "manifests/cluster-scheduler-02-config.yml"; +data = yaml.full_load(open(path)); +data["spec"]["mastersSchedulable"] = False; +open(path, "w").write(yaml.dump(data, default_flow_style=False))' +python3 -c ' +import yaml; +path = "manifests/cluster-dns-02-config.yml"; +data = yaml.full_load(open(path)); +del data["spec"]["publicZone"]; +del data["spec"]["privateZone"]; +open(path, "w").write(yaml.dump(data, default_flow_style=False))' +INFRA_ID=$(yq -r '.status.infrastructureName' manifests/cluster-infrastructure-02-config.yml) +RESOURCE_GROUP=$(yq -r '.status.platformStatus.azure.resourceGroupName' manifests/cluster-infrastructure-02-config.yml) +openshift-install create ignition-configs + +trap cleanupError 0 # Stop debug when failing +# set -x +echoDo "Create Resource Group" az group create --name $RESOURCE_GROUP --location $AZURE_REGION ${azout} +echoDo "Create Identity" az identity create -g $RESOURCE_GROUP -n ${INFRA_ID}-identity ${azout} +echoDo "Create Storage Account" az storage account create -g $RESOURCE_GROUP --location $AZURE_REGION --name ${CLUSTER_NAME}sa --kind Storage --sku Standard_LRS ${azout} +ACCOUNT_KEY=$(az storage account keys list -g $RESOURCE_GROUP --account-name ${CLUSTER_NAME}sa --query "[0].value" -o tsv) +VHD_URL=$(curl -s https://raw.githubusercontent.com/openshift/installer/release-$OCPRELEASE/data/data/rhcos.json | jq -r .azure.url) +echoDo "Create vhd Storage Container" az storage container create --name vhd --account-name ${CLUSTER_NAME}sa ${azout} +echoDo "Create file Storage Container" az storage container create --name files --account-name ${CLUSTER_NAME}sa --public-access blob ${azout} +echoDo "Upload bootstrap.ign to file storage" az storage blob upload --account-name ${CLUSTER_NAME}sa --account-key $ACCOUNT_KEY -c "files" -f "bootstrap.ign" -n "bootstrap.ign" ${azout} +echoDo "Start VHD to image creation" az storage blob copy start --account-name ${CLUSTER_NAME}sa --account-key $ACCOUNT_KEY --destination-blob "rhcos.vhd" --destination-container vhd --source-uri "$VHD_URL" +PRINCIPAL_ID=$(az identity show -g $RESOURCE_GROUP -n ${INFRA_ID}-identity --query principalId --out tsv) +RESOURCE_GROUP_ID=$(az group show -g $RESOURCE_GROUP --query id --out tsv) +echoDo "Create Service Account" az role assignment create --assignee "$PRINCIPAL_ID" --role 'Contributor' --scope "$RESOURCE_GROUP_ID" ${azout} +echoDo "Create public DNS Zone" az network dns zone create -g $RESOURCE_GROUP -n ${CLUSTER_NAME}.${BASE_DOMAIN} ${azout} +nsServer0=$(az network dns zone show --resource-group $RESOURCE_GROUP --name ${CLUSTER_NAME}.${BASE_DOMAIN} --query "nameServers[0]" -o tsv) +nsServer1=$(az network dns zone show --resource-group $RESOURCE_GROUP --name ${CLUSTER_NAME}.${BASE_DOMAIN} --query "nameServers[1]" -o tsv) +echoDo "Create NS Record Sets" az network dns record-set ns create --name ${CLUSTER_NAME} --resource-group $BASE_DOMAIN_RESOURCE_GROUP --zone-name ${BASE_DOMAIN} ${azout} +echoDo "Create NS record" az network dns record-set ns add-record --nsdname $nsServer0 -n ${CLUSTER_NAME} -g $BASE_DOMAIN_RESOURCE_GROUP -z ${BASE_DOMAIN} ${azout} +echoDo "Create NS record" az network dns record-set ns add-record --nsdname $nsServer1 -n ${CLUSTER_NAME} -g $BASE_DOMAIN_RESOURCE_GROUP -z ${BASE_DOMAIN} ${azout} +echoDo "Create Private DNS zone" az network private-dns zone create -g $RESOURCE_GROUP -n ${CLUSTER_NAME}.${BASE_DOMAIN} ${azout} +status="unknown" +while [ "$status" != "success" ] +do + status=$(az storage blob show --container-name vhd --name "rhcos.vhd" --account-name ${CLUSTER_NAME}sa --account-key $ACCOUNT_KEY -o tsv --query properties.copy.status) + echo $status +done +echoDo "Create VNET" az deployment group create -g $RESOURCE_GROUP \ + --template-file "01_vnet.json" \ + --parameters baseName="$INFRA_ID" ${azout} +echoDo "Associate private-dns with virtual network" az network private-dns link vnet create -g $RESOURCE_GROUP -z ${CLUSTER_NAME}.${BASE_DOMAIN} -n ${INFRA_ID}-network-link -v "${INFRA_ID}-vnet" -e false ${azout} +VHD_BLOB_URL=$(az storage blob url --account-name ${CLUSTER_NAME}sa --account-key $ACCOUNT_KEY -c vhd -n "rhcos.vhd" -o tsv) +# looks like we need a short delay here +sleep 10 +echoDo "Setup azure VHD storage" az deployment group create -g $RESOURCE_GROUP \ + --template-file "02_storage.json" \ + --parameters vhdBlobURL="$VHD_BLOB_URL" \ + --parameters baseName="$INFRA_ID" ${azout} +echoDo "Setup PrivateDNS records for infrastructure" az deployment group create -g $RESOURCE_GROUP \ + --template-file "03_infra.json" \ + --parameters privateDNSZoneName="${CLUSTER_NAME}.${BASE_DOMAIN}" \ + --parameters baseName="$INFRA_ID" ${azout} +PUBLIC_IP=$(az network public-ip list -g $RESOURCE_GROUP --query "[?name=='${INFRA_ID}-master-pip'] | [0].ipAddress" -o tsv) +echoDo "Add public A record for api end point" az network dns record-set a add-record -g $RESOURCE_GROUP -z ${CLUSTER_NAME}.${BASE_DOMAIN} -n api -a $PUBLIC_IP --ttl 60 ${azout} +BOOTSTRAP_URL=$(az storage blob url --account-name ${CLUSTER_NAME}sa --account-key $ACCOUNT_KEY -c "files" -n "bootstrap.ign" -o tsv) +BOOTSTRAP_IGNITION=$(jq -rcnM --arg v "2.2.0" --arg url $BOOTSTRAP_URL '{ignition:{version:$v,config:{replace:{source:$url}}}}' | base64 -w0) +echoDo "Create Bootstrap" az deployment group create -g $RESOURCE_GROUP \ + --template-file "04_bootstrap.json" \ + --parameters bootstrapIgnition="$BOOTSTRAP_IGNITION" \ + --parameters sshKeyData="$SSH_KEY" \ + --parameters baseName="$INFRA_ID" ${azout} +echoDo "Create masters" az deployment group create -g $RESOURCE_GROUP \ + --template-file "05_masters.json" \ + --parameters masterIgnition="$(base64 -w0 master.ign)" \ + --parameters sshKeyData="$SSH_KEY" \ + --parameters privateDNSZoneName="${CLUSTER_NAME}.${BASE_DOMAIN}" \ + --parameters baseName="$INFRA_ID" \ + --no-wait ${azout} +echoDo "Wait for OCP Bootstrap to complete" openshift-install wait-for bootstrap-complete --log-level debug +export KUBECONFIG="$PWD/auth/kubeconfig" +oc get nodes +oc get clusteroperator + +echoDo "Create Worker Nodes" az deployment group create -g $RESOURCE_GROUP \ + --template-file "06_workers.json" \ + --parameters workerIgnition="$(base64 -w0 worker.ign)" \ + --parameters sshKeyData="$SSH_KEY" \ + --parameters baseName="$INFRA_ID" \ + --parameters numberOfNodes="$WORKERNODES" \ + --no-wait ${azout} + +# Remove bootstrap +echo Remove Bootstrap resources +echoDo "Remove SSH access to Bootstrap" az network nsg rule delete -g $RESOURCE_GROUP --nsg-name ${INFRA_ID}-controlplane-nsg --name bootstrap_ssh_in ${azout} +echoDo "Stop Bootstrap VM" az vm stop -g $RESOURCE_GROUP --name ${INFRA_ID}-bootstrap ${azout} +echoDo "Deallocated Bootstrap VM" az vm deallocate -g $RESOURCE_GROUP --name ${INFRA_ID}-bootstrap ${azout} +echoDo "Delete Bootstrap VM" az vm delete -g $RESOURCE_GROUP --name ${INFRA_ID}-bootstrap --yes ${azout} +echoDo "Delete Bootstrap OS Disk" az disk delete -g $RESOURCE_GROUP --name ${INFRA_ID}-bootstrap_OSDisk --no-wait --yes ${azout} +echoDo "Delete Bootstrap NIC" az network nic delete -g $RESOURCE_GROUP --name ${INFRA_ID}-bootstrap-nic --no-wait ${azout} +echoDo "Delete ignition file for RHCOS bootstrapping" az storage blob delete --account-key $ACCOUNT_KEY --account-name ${CLUSTER_NAME}sa --container-name files --name bootstrap.ign ${azout} +echoDo "Delete public IP for ssh access" az network public-ip delete -g $RESOURCE_GROUP --name ${INFRA_ID}-bootstrap-ssh-pip ${azout} + +echo All done .... goodbye!