WiP: Support custom Azure environment installation #3529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

cfBrianMiller wants to merge 39 commits into openshift:master from CloudFitSoftware:azuremag/upi

docs/user/azure/README.md

-Original file line number
+Diff line change
@@ Expand Up @@
     Please report a new issue if you do not find an issue related to any trouble
     you're having.
+    ## Azure MAG
+    Please see [Azure MAG Instructions](install_upi_azuremag.md), this is currently in pre-alpha.
     [issues]: https://github.com/openshift/installer/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+azure

docs/user/azure/install_upi_azuremag.md

-Original file line number
+Diff line change
@@ -0,0 +1,181 @@
+    # Azure MAG UPI Instructions
+    This brief guide will demonstrate how to use the UPI installer to install OCP 4.X in Azure Government
+    ## Notice of supportability
+    CloudFit Software, partnering with Microsoft and Red Hat are actively working towards support exceptions that will cover OCP 4.3 implementations running on Azure Government implementations.  This information will be published and updated periodically.
+    Until there is a support excpetion Red Hat recommends that this only be used in proof of concept (POC) environments, and does not recommend this for production use cases as it will not be supported.
+    ## Install Procedures
+    ### Prerequisites
+    Applications in your $PATH
+    - Azcli
+    - jq
+    - yq
+    - oc client
+    - kubectl
+    - openshift-install
+    Azure Access
+    - Contributor access to an existing azure government subscription
+    - Ability to create the following resource types
+      - Disk
+      - DNS Zone
+      - Image
+      - Load Balancer
+      - Managed Identity
+      - Network Interface
+        - With a public ip binding (this will be resolved in a later release)
+      - Network Security Group
+      - Private DNS Zone
+      - Public IP address
+      - Storage Account
+      - Virtual Machine
+      - Virtual Network
+    Azure Rquirements
+    - Public DNS zone available in azure government
+    ### Create Install Config
+    Here is a sample install config, create a install-config.yaml file in an empty folder with this template to get started.
+    ```yaml
+    #install-config.yaml
+    apiVersion: v1
+    baseDomain: <YOUR_PUBLIC_DNS_ZONE>
+    compute:
+    - hyperthreading: Enabled
+      name: worker
+      platform: {}
+      replicas: 3
+    controlPlane:
+      hyperthreading: Enabled
+      name: master
+      platform: {}
+      replicas: 3
+    metadata:
+      creationTimestamp: null
+      name: <YOUR_CLUSTER_NAME>
+    networking:
+      clusterNetwork:
+      - cidr: 10.128.0.0/14
+        hostPrefix: 23
+      machineNetwork:
+      - cidr: 10.0.0.0/16
+      networkType: OpenShiftSDN
+      serviceNetwork:
+      - 172.30.0.0/16
+    platform:
+      azure:
+        baseDomainResourceGroupName: <YOUR_DNS_ZONE_RG>
+        region: <YOUR_AZURE_GOVERNMENT_REGION>
+    publish: External
+    pullSecret: '<YOUR_PULL_SECRET>'
+    sshKey: |
+     <YOUR_SSH_KEY>
+    ```
+    Create or modify this file to ensure the right azure environment secrets get passed, ~/.azure/osServicePrincipal.json. All of these values are in plain text, they will be translated to base64 encoded secrets during the install.
+    ```json
+    {
+      "subscriptionId":"<YOUR_AZURE_GOV_SUBSCRIPTION_ID",
+      "clientId":"<YOUR_AZURE_GOV_CLIENT_ID>",
+      "clientSecret":"<YOUR_AZURE_GOV_CLIENT_SECRET>",
+      "tenantId":"<YOUR_AZURE_GOV_TENANT_ID>"
+    }
+    ```
+    ### Copy required files
+    Copy the following files from $CODE_LOCATION/upi/azure
+    - 01_vnet.json
+    - 02_storage.json
+    - 03_infra.json
+    - 04_bootstrap.json
+    - 05_masters.json
+    - 06_workers.json
+    - azureGovQuickstart.sh
+    and make the azureGovQuickstart.sh file executable.
+    A quick way to do this is (assuming default code location)
+    ```shell
+    export CODE_LOCATION=~/go/src/github.com/openshift/installer
+    cp $CODE_LOCATION/upi/azure/0*.json ./
+    cp $CODE_LOCATION/upi/azure/azureGovQuickstart.sh ./
+    chmod +x azureGovQuickstart.sh
+    ```
+    ### Run Azure Gov Quickstart
+    Be in the directory with the copied files and run
+    ```shell
+    export WORKER_NODE_COUNT=<Number of workers you want, default of 3>
+    ./azureGovQuickstart.sh -w $WORKER_NODE_COUNT
+    ```
+    #### Log into azure portal
+    The installer will prompt you to sign into the azure gov portal with a web browser pop up.
+    #### Finish Cluster Configuration
+    Wait 10 minutes for the nodes to requst cluster membership and then run the following.
+    After the script completes configure kubeconfig, join the work nodes, edit the registry operator, and launch the web console.
+    ```bash
+    # Configure kubeconfig to authenticate against new openshift
+    export KUBECONFIG="$PWD/auth/kubeconfig"
+    # Get CSR ids of pending requests
+    oc get csr -A
+    # Approve CSR ids
+    oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve
+    # Nodes should populate and become ready in a couple of minutes
+    watch oc get nodes
+    # Edit image registry operator config
+    # The storage operator doesn't yet work with Azure Mag so the internal registry has to be disabled
+    oc edit configs.imageregistry.operator.openshift.io/cluster
+    # change managedState: Managed --> Removed
+    ```
+    Add DNS Records in public and private dns zone of the ip address assigned to the new load balancer.
+    *.apps --> ip address of new LB
+    ```bash
+    # Complete the cluster install and get temporary admin password for web console
+    openshift-install wait-for install-complete
+    ```
+    #### Add Azure Disk Storage Class
+    Add the following storage class to your deployment, for more information see official [docs](https://kubernetes.io/docs/concepts/storage/storage-classes/#azure-disk-storage-class)
+    ```yaml
+    # azure-disk.yaml
+    apiVersion: storage.k8s.io/v1
+    kind: StorageClass
+    metadata:
+      annotations:
+        description: azure disk
+        storageclass.kubernetes.io/is-default-class: "true"
+      name: azuredisk
+    parameters:
+      kind: managed
+      location: <your_region>
+      skuName: <your_sku>
+    provisioner: kubernetes.io/azure-disk
+    reclaimPolicy: Delete
+    volumeBindingMode: Immediate
+    ```

docs/user/azure/install_upi_azuremag_quickstart.md

-Original file line number
+Diff line change
@@ -0,0 +1,31 @@
+    # Azure MAG UPI quickstart
+. run ```openshift-install create install-config```
+        - input valid azure public information (this won't be used)
+. Modify install-config with the correct values for azure government
+        - platform.azure.region
+        - platform.azure.baseDomainResourceGroupName
+        - basedomain
+. copy all ARM templates from $CODE_LOCATION/upi/azure
+. copy azureGovQuickstart.sh from $CODE_LOCATION/upi/azure
+. run azureGovQuickstart.sh -w {Number of desired worker nodes}
+. when installer pauses edit the following files
+. manifests/cloud-provider-config with the correct information
+          - cloud: AzurePublicCloud --> AzureUSGovernmentCloud
+          - tenantId: {Azure Public tenant} --> Azure Gov Tenant
+          - subscriptionId: {Azure Public Subscription} --> Azure Gov Subscription
+. openshift/99_cloud-creds-secret with the correct information
+          - azure_subscription_id: {base64 encoded Public subscription} --> base64 Azure Gov Subscription
+          - azure_client_id: {base64 encoded Client ID} --> base64 Azure Gov client id
+          - azure_client_secret: {base64 encoded Client Secret} --> base64 Azure Gov Client Secret
+          - azure_tenant_id: {base64 encoded tenant id} --> base64 Azure Gov tenant id
+. run ```export KUBECONFIG="$PWD/auth/kubeconfig"```
+. watch for incoming CSRs with ```watch oc get csr -A```
+        - will look like this "system:serviceaccount:openshift-machine-config-operator:node-bootstrapper"
+. Approve CSRs with ```oc adm certificate approve [csr IDs]```
+. after CSRs have been approved you should have 3 master nodes and desired worker nodes ```oc get nodes```
+. run ```oc edit configs.imageregistry.operator.openshift.io/cluster ```
+        - change managedState: Managed --> Removed
+. add A records to the public and private dns zone for the created node LB
+        - *.apps.{base domain}
+. run ```openshift-install wait-for install-complete``` for the username and password of the web console

pkg/asset/installconfig/azure/azure.go

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -16,7 +16,7 @@ import (
  
    )

    const (

    	defaultRegion string = "eastus"

    	azureEnvironment string = "AZURE_ENVIRONMENT"

    )

    // Platform collects azure-specific configuration.

    @@ -47,11 +47,6 @@ func Platform() (*azure.Platform, error) {
  
    		return strings.SplitN(s, " ", 2)[0]

    	})

    	_, ok := regions[defaultRegion]

    	if !ok {

    		return nil, errors.Errorf("installer bug: invalid default azure region %q", defaultRegion)

    	}

    	sort.Strings(longRegions)

    	sort.Strings(shortRegions)

    @@ -61,7 +56,6 @@ func Platform() (*azure.Platform, error) {
  
    			Prompt: &survey.Select{

    				Message: "Region",

    				Help:    "The azure region to be used for installation.",

    				Default: fmt.Sprintf("%s (%s)", defaultRegion, regions[defaultRegion]),

    				Options: longRegions,

    			},

    			Validate: survey.ComposeValidators(survey.Required, func(ans interface{}) error {

    @@ -89,7 +83,8 @@ func getRegions() (map[string]string, error) {
  
    	if err != nil {

    		return nil, err

    	}

    	client := azsub.NewClientWithBaseURI(session.Environment.ResourceManagerEndpoint)

    	client.Authorizer = session.Authorizer

    	ctx, cancel := context.WithTimeout(context.TODO(), 1*time.Minute)

    	defer cancel()

    @@ -110,7 +105,8 @@ func getResourceCapableRegions() ([]string, error) {
  
    	if err != nil {

    		return nil, err

    	}

    	client := azres.NewProvidersClientWithBaseURI(session.Environment.ResourceManagerEndpoint, session.Credentials.SubscriptionID)

    	client.Authorizer = session.Authorizer

    	ctx, cancel := context.WithTimeout(context.TODO(), 1*time.Minute)

    	defer cancel()

pkg/asset/installconfig/azure/dns.go

-Original file line number
+Diff line change
@@ Expand Up / @@ -118,7 +118,7 @@ func NewDNSConfig() (*DNSConfig, error) { @@
     }
     func newZonesClient(session *Session) ZonesGetter {
-    	azureClient := azdns.NewZonesClient(session.Credentials.SubscriptionID)
+    	azureClient := azdns.NewZonesClientWithBaseURI(session.Environment.ResourceManagerEndpoint, session.Credentials.SubscriptionID)
     	azureClient.Authorizer = session.Authorizer
     	return &ZonesClient{azureClient: azureClient}
     }
@@ Expand Down @@

pkg/asset/installconfig/azure/session.go

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -20,13 +20,15 @@ const azureAuthEnv = "AZURE_AUTH_LOCATION"
  
    var (

    	defaultAuthFilePath = filepath.Join(os.Getenv("HOME"), ".azure", "osServicePrincipal.json")

    	onceLoggers         = map[string]*sync.Once{}

    	defaultAzureEnvironment = azureenv.PublicCloud

    )

    //Session is an object representing session for subscription

    type Session struct {

    	GraphAuthorizer autorest.Authorizer

    	Authorizer      autorest.Authorizer

    	Credentials     Credentials

    	Environment     azureenv.Environment

    }

    //Credentials is the data type for credentials as understood by the azure sdk

    @@ -37,21 +39,37 @@ type Credentials struct {
  
    	TenantID       string `json:"tenantId,omitempty"`

    }

    func getEnvironmentFromOS() azureenv.Environment {

    	env := defaultAzureEnvironment

    	if envSet := os.Getenv(azureEnvironment); len(envSet) > 0 {

    		logrus.Debugf("Found Azure environment override, trying to set environment to %v", envSet)

    		foundEnvironment, err := azureenv.EnvironmentFromName(envSet)

    		if err == nil {

    			env = foundEnvironment

    			logrus.Debugf("Successfully set azure environment to %v", foundEnvironment.Name)

    		}else {

    			logrus.Debugf("Error setting environment, %v", err.Error())

    		}

    	}

    	return env

    }

    // GetSession returns an azure session by using credentials found in ~/.azure/osServicePrincipal.json

    // and, if no creds are found, asks for them and stores them on disk in a config file

    func GetSession() (*Session, error) {

    	authFile := defaultAuthFilePath

    	if f := os.Getenv(azureAuthEnv); len(f) > 0 {

    		authFile = f

    	}

    	return newSessionFromFile(authFile)

    	env := getEnvironmentFromOS()

    	return newSessionFromFile(authFile, env)

    }

    func newSessionFromFile(authFilePath string) (*Session, error) {

    func newSessionFromFile(authFilePath string, env azureenv.Environment) (*Session, error) {

    	// NewAuthorizerFromFileWithResource uses `auth.GetSettingsFromFile`, which uses the `azureAuthEnv` to fetch the auth credentials.

    	// therefore setting the local env here to authFilePath allows NewAuthorizerFromFileWithResource to load credentials.

    	os.Setenv(azureAuthEnv, authFilePath)

    	_, err := auth.NewAuthorizerFromFileWithResource(azureenv.PublicCloud.ResourceManagerEndpoint)

    	_, err := auth.NewAuthorizerFromFileWithResource(env.ResourceManagerEndpoint)

    	if err != nil {

    		logrus.Debug("Could not get an azure authorizer from file. Asking user to provide authentication info")

    		credentials, err := askForCredentials()

    @@ -82,12 +100,12 @@ func newSessionFromFile(authFilePath string) (*Session, error) {
  
    		logrus.Infof("Credentials loaded from file %q", authFilePath)

    	})

    	authorizer, err := authSettings.ClientCredentialsAuthorizerWithResource(azureenv.PublicCloud.ResourceManagerEndpoint)

    	authorizer, err := authSettings.ClientCredentialsAuthorizerWithResource(env.ResourceManagerEndpoint)

    	if err != nil {

    		return nil, errors.Wrap(err, "failed to get client credentials authorizer from saved azure auth settings")

    	}

    	graphAuthorizer, err := authSettings.ClientCredentialsAuthorizerWithResource(azureenv.PublicCloud.GraphEndpoint)

    	graphAuthorizer, err := authSettings.ClientCredentialsAuthorizerWithResource(env.GraphEndpoint)

    	if err != nil {

    		return nil, errors.Wrap(err, "failed to get GraphEndpoint authorizer from saved azure auth settings")

    	}

    @@ -96,6 +114,7 @@ func newSessionFromFile(authFilePath string) (*Session, error) {
  
    		GraphAuthorizer: graphAuthorizer,

    		Authorizer:      authorizer,

    		Credentials:     *credentials,

    		Environment:     env,

    	}, nil

    }

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WiP: Support custom Azure environment installation #3529

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!