Add VSphere IPI e2e steps #7586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

openshift-merge-robot merged 2 commits into openshift:master from patrickdillon:vsphere-ipi-steps

Mar 25, 2020

ci-operator/config/openshift/installer/openshift-installer-master.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -202,3 +202,7 @@ tests: @@
       steps:
         cluster_profile: packet
         workflow: baremetalds-e2e
+    - as: e2e-vsphere-steps
+      steps:
+        cluster_profile: vsphere
+        workflow: origin-e2e-vsphere

ci-operator/jobs/openshift/installer/openshift-installer-master-presubmits.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -2269,6 +2269,83 @@ presubmits: @@
             secret:
               secretName: sentry-dsn
         trigger: (?m)^/test( | .* )e2e-vsphere,?($|\s.*)
+      - agent: kubernetes
+        always_run: false
+        branches:
+        - master
+        cluster: api.ci
+        context: ci/prow/e2e-vsphere-steps
+        decorate: true
+        decoration_config:
+          skip_cloning: true
+        labels:
+          ci-operator.openshift.io/prowgen-controlled: "true"
+          pj-rehearse.openshift.io/can-be-rehearsed: "true"
+        name: pull-ci-openshift-installer-master-e2e-vsphere-steps
+        optional: true
+        rerun_command: /test e2e-vsphere-steps
+        spec:
+          containers:
+          - args:
+            - --artifact-dir=$(ARTIFACTS)
+            - --give-pr-author-access-to-namespace=true
+            - --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
+            - --kubeconfig=/etc/apici/kubeconfig
+            - --lease-server-password-file=/etc/boskos/password
+            - --lease-server-username=ci
+            - --lease-server=https://boskos-ci.svc.ci.openshift.org
+            - --secret-dir=/usr/local/e2e-vsphere-steps-cluster-profile
+            - --sentry-dsn-path=/etc/sentry-dsn/ci-operator
+            - --target=e2e-vsphere-steps
+            command:
+            - ci-operator
+            image: ci-operator:latest
+            imagePullPolicy: Always
+            name: ""
+            resources:
+              requests:
+                cpu: 10m
+            volumeMounts:
+            - mountPath: /etc/apici
+              name: apici-ci-operator-credentials
+              readOnly: true
+            - mountPath: /etc/boskos
+              name: boskos
+              readOnly: true
+            - mountPath: /usr/local/e2e-vsphere-steps-cluster-profile
+              name: cluster-profile
+            - mountPath: /etc/pull-secret
+              name: pull-secret
+              readOnly: true
+            - mountPath: /etc/sentry-dsn
+              name: sentry-dsn
+              readOnly: true
+          serviceAccountName: ci-operator
+          volumes:
+          - name: apici-ci-operator-credentials
+            secret:
+              items:
+              - key: sa.ci-operator.apici.config
+                path: kubeconfig
+              secretName: apici-ci-operator-credentials
+          - name: boskos
+            secret:
+              items:
+              - key: password
+                path: password
+              secretName: boskos-credentials
+          - name: cluster-profile
+            projected:
+              sources:
+              - secret:
+                  name: cluster-secrets-vsphere
+          - name: pull-secret
+            secret:
+              secretName: regcred
+          - name: sentry-dsn
+            secret:
+              secretName: sentry-dsn
+        trigger: (?m)^/test( | .* )e2e-vsphere-steps,?($|\s.*)
       - agent: kubernetes
         always_run: true
         branches:
@@ Expand Down @@

ci-operator/step-registry/ipi/conf/vsphere/OWNERS

-Original file line number
+Diff line change
@@ -0,0 +1,4 @@
+    approvers:
+    - abhinavdahiya
+    - jcpowermac
+    - patrickdillon

ci-operator/step-registry/ipi/conf/vsphere/dns/OWNERS

-Original file line number
+Diff line change
@@ -0,0 +1,4 @@
+    approvers:
+    - abhinavdahiya
+    - jcpowermac
+    - patrickdillon

ci-operator/step-registry/ipi/conf/vsphere/dns/ipi-conf-vsphere-dns-commands.sh

-Original file line number
+Diff line change
@@ -0,0 +1,92 @@
+    #!/bin/bash
+    set -o nounset
+    set -o errexit
+    set -o pipefail
+    echo "origin-ci-int-aws.dev.rhcloud.com" > "${SHARED_DIR}"/basedomain.txt
+    cluster_profile=/var/run/secrets/ci.openshift.io/cluster-profile
+    cluster_name=${NAMESPACE}-${JOB_NAME_HASH}
+    base_domain=$(<"${SHARED_DIR}"/basedomain.txt)
+    cluster_domain="${cluster_name}.${base_domain}"
+    export AWS_SHARED_CREDENTIALS_FILE=${cluster_profile}/.awscred
+    # Load array created in setup-vips:
+    # 0: API
+    # 1: Ingress
+    # 2: DNS
+    declare -a vips
+    mapfile -t vips < "${SHARED_DIR}"/vips.txt
+    hosted_zone_id="$(aws route53 list-hosted-zones-by-name \
+                --dns-name "${base_domain}" \
+                --query "HostedZones[? Config.PrivateZone != \`true\` && Name == \`${base_domain}.\`].Id" \
+                --output text)"
+    echo "Creating DNS records..."
+    cat > "${SHARED_DIR}"/dns-create.json <<EOF
+    {
+    "Comment": "Create public OpenShift DNS records for VSphere IPI CI install",
+    "Changes": [{
+        "Action": "UPSERT",
+        "ResourceRecordSet": {
+          "Name": "api.$cluster_domain.",
+          "Type": "A",
+          "TTL": 60,
+          "ResourceRecords": [{"Value": "${vips[0]}"}]
+          }
+        },{
+        "Action": "UPSERT",
+        "ResourceRecordSet": {
+          "Name": "*.apps.$cluster_domain.",
+          "Type": "A",
+          "TTL": 60,
+          "ResourceRecords": [{"Value": "${vips[1]}"}]
+          }
+        },{
+        "Action": "UPSERT",
+        "ResourceRecordSet": {
+          "Name": "$cluster_domain.",
+          "Type": "NS",
+          "TTL": 300,
+          "ResourceRecords": [{"Value": "${vips[2]}"}]
+          }
+    }]}
+    EOF
+    aws route53 change-resource-record-sets --hosted-zone-id "$hosted_zone_id" --change-batch file:///"${SHARED_DIR}"/dns-create.json
+    echo "Creating batch file to destroy DNS records"
+    cat > "${SHARED_DIR}"/dns-delete.json <<EOF
+    {
+    "Comment": "Delete public OpenShift DNS records for VSphere IPI CI install",
+    "Changes": [{
+        "Action": "DELETE",
+        "ResourceRecordSet": {
+          "Name": "api.$cluster_domain.",
+          "Type": "A",
+          "TTL": 60,
+          "ResourceRecords": [{"Value": "${vips[0]}"}]
+          }
+        },{
+        "Action": "DELETE",
+        "ResourceRecordSet": {
+          "Name": "*.apps.$cluster_domain.",
+          "Type": "A",
+          "TTL": 60,
+          "ResourceRecords": [{"Value": "${vips[1]}"}]
+          }
+        },{
+        "Action": "DELETE",
+        "ResourceRecordSet": {
+          "Name": "$cluster_domain.",
+          "Type": "NS",
+          "TTL": 300,
+          "ResourceRecords": [{"Value": "${vips[2]}"}]
+          }
+    }]}
+    EOF

ci-operator/step-registry/ipi/conf/vsphere/dns/ipi-conf-vsphere-dns-ref.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,13 @@
+    ref:
+      as: ipi-conf-vsphere-dns
+      from: upi-installer
+      commands: ipi-conf-vsphere-dns-commands.sh
+      resources:
+        requests:
+          cpu: 10m
+          memory: 100Mi
+      documentation: >-
+        Uses VIPs in ${SHARED_DIR}/vips.txt to create route53 dns records. Outputs
+        ${SHARED_DIR}/basedomain.txt to ensure consistent basedomain in conf and
+        deprovision steps. Saves batch job to delete DNS records to
+        ${SHARED_DIR}/dns-delete.json for use in deprovisioning.

ci-operator/step-registry/ipi/conf/vsphere/ipi-conf-vsphere-chain.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,11 @@
+    chain:
+      as: ipi-conf-vsphere
+      steps:
+      - ref: ipi-conf-vsphere-vips
+      - ref: ipi-conf-vsphere-dns
+      - ref: ipi-conf
+      - ref: ipi-conf-vsphere
+      documentation: >-
+        The vSphere IPI configure step chain generates prerequisites for installing
+        a cluster: virtual IP addresses, DNS records, and the install-config.yaml.
+        Resources are created based on the cluster profile and optional input files.

ci-operator/step-registry/ipi/conf/vsphere/ipi-conf-vsphere-commands.sh

-Original file line number
+Diff line change
@@ -0,0 +1,30 @@
+    #!/bin/bash
+    set -o nounset
+    set -o errexit
+    set -o pipefail
+    CONFIG="${SHARED_DIR}/install-config.yaml"
+    TFVARS_PATH=/var/run/secrets/ci.openshift.io/cluster-profile/secret.auto.tfvars
+    vsphere_user=$(grep -oP 'vsphere_user="\K[^"]+' ${TFVARS_PATH})
+    vsphere_password=$(grep -oP 'vsphere_password="\K[^"]+' ${TFVARS_PATH})
+    base_domain=$(<"${SHARED_DIR}"/basedomain.txt)
+    declare -a vips
+    mapfile -t vips < "${SHARED_DIR}/vips.txt"
+    cat >> "${CONFIG}" << EOF
+    baseDomain: $base_domain
+    platform:
+      vsphere:
+        cluster: devel
+        datacenter: dc1
+        defaultDatastore: nvme-ds1
+        network: VM Network
+        password: ${vsphere_password}
+        username: ${vsphere_user}
+        vCenter: vcsa-ci.vmware.devcluster.openshift.com
+        apiVIP: "${vips[0]}"
+        ingressVIP: "${vips[1]}"
+        dnsVIP: "${vips[2]}"
+    EOF

ci-operator/step-registry/ipi/conf/vsphere/ipi-conf-vsphere-ref.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,12 @@
+    ref:
+      as: ipi-conf-vsphere
+      from: base
+      commands: ipi-conf-vsphere-commands.sh
+      resources:
+        requests:
+          cpu: 10m
+          memory: 100Mi
+      documentation: >-
+        The IPI vSphere configure step generates the vSphere-specific
+        install-config.yaml contents based on the cluster profile,
+        $SHARED_DIR/vips.txt, $SHARED_DIR/basedomain.txt, and optional input files.

ci-operator/step-registry/ipi/conf/vsphere/vips/OWNERS

-Original file line number
+Diff line change
@@ -0,0 +1,4 @@
+    approvers:
+    - abhinavdahiya
+    - jcpowermac
+    - patrickdillon

ci-operator/step-registry/ipi/conf/vsphere/vips/ipi-conf-vsphere-vips-commands.sh

-Original file line number
+Diff line change
@@ -0,0 +1,35 @@
+    #!/bin/bash
+    set -o nounset
+    set -o errexit
+    set -o pipefail
+    tfvars_path=/var/run/secrets/ci.openshift.io/cluster-profile/secret.auto.tfvars
+    cluster_name=${NAMESPACE}-${JOB_NAME_HASH}
+    ipam_token=$(grep -oP 'ipam_token="\K[^"]+' ${tfvars_path})
+    # Array to hold virtual ips:
+    # 0: API
+    # 1: Ingress
+    # 2: DNS
+    declare -a vips
+    echo "Reserving virtual ip addresses from the IPAM server..."
+    for i in {0..2}
+    do
+      args=$(jq -n \
+                --arg hostn "$cluster_name-$i" \
+                --arg token "$ipam_token" \
+                '{network: "139.178.87.128", hostname: $hostn, ipam: "139.178.89.254", ipam_token: $token}')
+      vip_json=$(echo "$args" | bash <(curl -s https://raw.githubusercontent.com/openshift/installer/master/upi/vsphere/machine/cidr_to_ip.sh))
+      vips[$i]=$(echo "$vip_json" | jq -r .ip_address )
+      if [[ -z ${vips[$i]} ]]; then
+        echo "error: Unable to reserve virtual IP address, exiting" 1>&2
+        exit 1
+      fi
+      echo "${vips[$i]}" >> "${SHARED_DIR}"/vips.txt
+    done
+    echo "Reserved the following IP addresses..."
+    cat "${SHARED_DIR}"/vips.txt

ci-operator/step-registry/ipi/conf/vsphere/vips/ipi-conf-vsphere-vips-ref.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,13 @@
+    ref:
+      as: ipi-conf-vsphere-vips
+      from: upi-installer
+      commands: ipi-conf-vsphere-vips-commands.sh
+      resources:
+        requests:
+          cpu: 10m
+          memory: 100Mi
+      documentation: >-
+        Using secrets from the vSphere cluster profile, the vSphere VIP setup step
+        reserves IP addresses through IPAM and saves them to $SHARED_DIR/vips.txt
+        for later use in creating DNS records and the install config. They are also
+        needed in deprovisioning to release the reserved VIPs.

ci-operator/step-registry/ipi/deprovision/vsphere/OWNERS

-Original file line number
+Diff line change
@@ -0,0 +1,4 @@
+    approvers:
+    - abhinavdahiya
+    - jcpowermac
+    - patrickdillon

ci-operator/step-registry/ipi/deprovision/vsphere/ipi-deprovision-vsphere-chain.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,7 @@
+    chain:
+      as: ipi-deprovision-vsphere
+      steps:
+      - chain: gather
+      - ref: ipi-deprovision-vsphere
+      documentation: |-
+        The IPI deprovision step chain contains all the individual steps necessary to deprovision an OpenShift cluster.

ci-operator/step-registry/ipi/deprovision/vsphere/ipi-deprovision-vsphere-commands.sh

-Original file line number
+Diff line change
@@ -0,0 +1,28 @@
+    #!/bin/bash
+    cluster_profile=/var/run/secrets/ci.openshift.io/cluster-profile
+    tfvars_path=/var/run/secrets/ci.openshift.io/cluster-profile/secret.auto.tfvars
+    base_domain=$(<"${SHARED_DIR}"/basedomain.txt)
+    cluster_name=${NAMESPACE}-${JOB_NAME_HASH}
+    ipam_token=$(grep -oP 'ipam_token="\K[^"]+' ${tfvars_path})
+    export AWS_SHARED_CREDENTIALS_FILE=${cluster_profile}/.awscred
+    echo "Deprovisioning cluster ..."
+    cp -ar "${SHARED_DIR}" /tmp/installer
+    TF_LOG=debug openshift-install --dir /tmp/installer destroy cluster
+    cp /tmp/installer/.openshift_install.log "${ARTIFACT_DIR}/"
+    hosted_zone_id="$(aws route53 list-hosted-zones-by-name \
+                --dns-name "${base_domain}" \
+                --query "HostedZones[? Config.PrivateZone != \`true\` && Name == \`${base_domain}.\`].Id" \
+                --output text)"
+    echo "Releasing IP addresses from IPAM server..."
+    for i in {0..2}
+    do
+        curl -s "http://139.178.89.254/api/removeHost.php?apiapp=address&apitoken=${ipam_token}&host=${cluster_name}-$i"
+    done
+    echo "Deleting Route53 DNS records..."
+    aws route53 change-resource-record-sets --hosted-zone-id "$hosted_zone_id" --change-batch file:///"${SHARED_DIR}"/dns-delete.json

ci-operator/step-registry/ipi/deprovision/vsphere/ipi-deprovision-vsphere-ref.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,12 @@
+    ref:
+      as: ipi-deprovision-vsphere
+      from: upi-installer
+      commands: ipi-deprovision-vsphere-commands.sh
+      resources:
+        requests:
+          cpu: 1000m
+          memory: 100Mi
+      documentation: >-
+        Reads the VIP and DNS records created in the setup phases from $SHARED_DIR
+        and delete them. Also uses $SHARED_DIR/basedomain.txt to ensure consistent
+        basedomain as setup steps.

ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh

-Original file line number
+Diff line change
@@ Expand Up / @@ -17,6 +17,7 @@ case "${CLUSTER_TYPE}" in @@
     aws) export AWS_SHARED_CREDENTIALS_FILE=${cluster_profile}/.awscred;;
     azure4) export AZURE_AUTH_LOCATION=${cluster_profile}/osServicePrincipal.json;;
     gcp) export GOOGLE_CLOUD_KEYFILE_JSON=${cluster_profile}/gce.json;;
+    vsphere) ;;
     *) echo >&2 "Unsupported cluster type '${CLUSTER_TYPE}'"
     esac
@@ Expand Down @@

ci-operator/step-registry/ipi/install/vsphere/OWNERS

-Original file line number
+Diff line change
@@ -0,0 +1,4 @@
+    approvers:
+    - abhinavdahiya
+    - jcpowermac
+    - patrickdillon

ci-operator/step-registry/ipi/install/vsphere/ipi-install-vsphere-chain.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,7 @@
+    chain:
+      as: ipi-install-vsphere
+      steps:
+      - ref: ipi-install-rbac
+      - ref: ipi-install-vsphere
+      documentation: |-
+        The IPI install step chain contains all the individual steps necessary to install an OpenShift cluster.

ci-operator/step-registry/ipi/install/vsphere/ipi-install-vsphere-commands.sh

Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		../install/ipi-install-install-commands.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add VSphere IPI e2e steps #7586

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!