vSphere UPI: Migrate to steps #9552
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| approvers: | ||
| - abhinavdahiya | ||
| - jcpowermac | ||
| - patrickdillon |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| workflow: | ||
| as: openshift-e2e-vsphere-upi | ||
| steps: | ||
| pre: | ||
| - chain: upi-vsphere-pre | ||
| test: | ||
| - ref: openshift-e2e-test | ||
| post: | ||
| - chain: upi-vsphere-post | ||
| documentation: |- | ||
| VSphere E2E wraps the Openshift E2E workflow and executes the common end-to-end test suite. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| approvers: | ||
| - abhinavdahiya | ||
| - jcpowermac | ||
| - patrickdillon |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| approvers: | ||
| - abhinavdahiya | ||
| - jcpowermac | ||
| - patrickdillon |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| #!/bin/bash | ||
|
|
||
| set -o nounset | ||
| set -o errexit | ||
| set -o pipefail | ||
|
|
||
| trap 'CHILDREN=$(jobs -p); if test -n "${CHILDREN}"; then kill ${CHILDREN} && wait; fi' TERM | ||
|
|
||
| HOME=/tmp | ||
| export HOME | ||
|
|
||
| echo "$(date -u --rfc-3339=seconds) - Locating RHCOS image for release..." | ||
|
|
||
| openshift_install_path="/var/lib/openshift-install" | ||
| image_json_file="${openshift_install_path}/rhcos.json" | ||
| fcos_json_file="${openshift_install_path}/fcos.json" | ||
|
|
||
| if [[ -f "$fcos_json_file" ]]; then | ||
| image_json_file=$fcos_json_file | ||
| fi | ||
|
|
||
| ova_url="$(jq -r '.baseURI + .images["vmware"].path' $image_json_file)" | ||
| vm_template="${ova_url##*/}" | ||
|
|
||
| echo "$(date -u --rfc-3339=seconds) - Configuring govc exports..." | ||
| # shellcheck source=/dev/null | ||
| source "${SHARED_DIR}/govc.sh" | ||
|
|
||
| cat > /tmp/rhcos.json << EOF | ||
| { | ||
| "DiskProvisioning": "thin", | ||
| "MarkAsTemplate": false, | ||
| "PowerOn": false, | ||
| "InjectOvfEnv": false, | ||
| "WaitForIP": false, | ||
| "Name": "${vm_template}" | ||
| } | ||
| EOF | ||
|
|
||
| echo "$(date -u --rfc-3339=seconds) - Checking if RHCOS OVA needs to be downloaded from ${ova_url}..." | ||
|
|
||
| if [[ "$(govc vm.info "${vm_template}" | wc -c)" -eq 0 ]] | ||
| then | ||
| echo "$(date -u --rfc-3339=seconds) - Creating a template for the VMs from ${ova_url}..." | ||
| curl -L -o /tmp/rhcos.ova "${ova_url}" | ||
| govc import.ova -options=/tmp/rhcos.json /tmp/rhcos.ova & | ||
| wait "$!" | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| ref: | ||
| as: upi-conf-vsphere-ova | ||
| from: upi-installer | ||
| commands: upi-conf-vsphere-ova-commands.sh | ||
| resources: | ||
| requests: | ||
| cpu: 1000m | ||
| memory: 512Mi | ||
| documentation: >- | ||
| vSphere OVA checks if appropriate template exists. | ||
| If it does not exist download the ova and upload to vCenter. | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| chain: | ||
| as: upi-conf-vsphere | ||
| steps: | ||
| - ref: ipi-conf | ||
| - ref: upi-conf-vsphere | ||
| - ref: upi-conf-vsphere-ova | ||
| documentation: >- | ||
| This chain generates the files necessary to deploy a vSphere UPI cluster. | ||
| It creates an install-config.yaml using the ipi-conf-vsphere chain. | ||
| Then it uses that file to create the Ignition, metadata.json, and kubeconfig files. | ||
| These files are copied to the shared dir for future consumption. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| #!/bin/bash | ||
|
|
||
| set -o nounset | ||
| set -o errexit | ||
| set -o pipefail | ||
|
|
||
| trap 'CHILDREN=$(jobs -p); if test -n "${CHILDREN}"; then kill ${CHILDREN} && wait; fi' TERM | ||
|
|
||
| if [[ -z "$RELEASE_IMAGE_LATEST" ]]; then | ||
| echo "RELEASE_IMAGE_LATEST is an empty string, exiting" | ||
| exit 1 | ||
| fi | ||
|
|
||
| export HOME=/tmp | ||
| export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${RELEASE_IMAGE_LATEST} | ||
| # Ensure ignition assets are configured with the correct invoker to track CI jobs. | ||
| export OPENSHIFT_INSTALL_INVOKER=openshift-internal-ci/${JOB_NAME_SAFE}/${BUILD_ID} | ||
|
|
||
| echo "$(date -u --rfc-3339=seconds) - Creating reusable variable files..." | ||
| # Create basedomain.txt | ||
| echo "origin-ci-int-aws.dev.rhcloud.com" > "${SHARED_DIR}"/basedomain.txt | ||
| base_domain=$(<"${SHARED_DIR}"/basedomain.txt) | ||
|
|
||
| # Create clustername.txt | ||
| echo "${NAMESPACE}-${JOB_NAME_HASH}" > "${SHARED_DIR}"/clustername.txt | ||
| cluster_name=$(<"${SHARED_DIR}"/clustername.txt) | ||
|
|
||
| # Create clusterdomain.txt | ||
| echo "${cluster_name}.${base_domain}" > "${SHARED_DIR}"/clusterdomain.txt | ||
| cluster_domain=$(<"${SHARED_DIR}"/clusterdomain.txt) | ||
jcpowermac marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
|
|
||
| ssh_pub_key_path="${CLUSTER_PROFILE_DIR}/ssh-publickey" | ||
| install_config="${SHARED_DIR}/install-config.yaml" | ||
| tfvars_path=/var/run/secrets/ci.openshift.io/cluster-profile/secret.auto.tfvars | ||
| vsphere_user=$(grep -oP 'vsphere_user\s*=\s*"\K[^"]+' ${tfvars_path}) | ||
| vsphere_password=$(grep -oP 'vsphere_password\s*=\s*"\K[^"]+' ${tfvars_path}) | ||
| ova_url="$(jq -r '.baseURI + .images["vmware"].path' /var/lib/openshift-install/rhcos.json)" | ||
| vm_template="${ova_url##*/}" | ||
|
|
||
|
|
||
| echo "$(date -u --rfc-3339=seconds) - Creating govc.sh file..." | ||
| cat >> "${SHARED_DIR}/govc.sh" << EOF | ||
| export GOVC_URL=vcsa-ci.vmware.devcluster.openshift.com | ||
| export GOVC_USERNAME="${vsphere_user}" | ||
| export GOVC_PASSWORD="${vsphere_password}" | ||
| export GOVC_INSECURE=1 | ||
| export GOVC_DATACENTER=dc1 | ||
| export GOVC_DATASTORE=vsanDatastore | ||
| EOF | ||
jcpowermac marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| echo "$(date -u --rfc-3339=seconds) - Extend install-config.yaml ..." | ||
|
|
||
| # We are not setting | ||
| # machineCIDR: 139.178.94.128/25 | ||
| # does this cause an issue? | ||
| cat >> "${install_config}" << EOF | ||
| baseDomain: $base_domain | ||
| controlPlane: | ||
| name: "master" | ||
| replicas: 3 | ||
| compute: | ||
| - name: "worker" | ||
| replicas: 0 | ||
| platform: | ||
| vsphere: | ||
| cluster: devel | ||
| datacenter: dc1 | ||
| defaultDatastore: vsanDatastore | ||
| network: "VM Network" | ||
| password: ${vsphere_password} | ||
| username: ${vsphere_user} | ||
| vCenter: vcsa-ci.vmware.devcluster.openshift.com | ||
| folder: "/dc1/vm/${cluster_name}" | ||
| EOF | ||
|
|
||
| echo "$(date -u --rfc-3339=seconds) - Create terraform.tfvars ..." | ||
| cat > "${SHARED_DIR}/terraform.tfvars" <<-EOF | ||
| machine_cidr = "139.178.94.128/25" | ||
| vm_template = "${vm_template}" | ||
| vsphere_cluster = "devel" | ||
| vsphere_datacenter = "dc1" | ||
| vsphere_datastore = "vsanDatastore" | ||
| vsphere_server = "vcsa-ci.vmware.devcluster.openshift.com" | ||
| ipam = "139.178.89.254" | ||
| cluster_id = "${cluster_name}" | ||
| base_domain = "${base_domain}" | ||
| cluster_domain = "${cluster_domain}" | ||
| ssh_public_key_path = "${ssh_pub_key_path}" | ||
| compute_memory = "16384" | ||
| compute_num_cpus = "4" | ||
| EOF | ||
|
|
||
| dir=/tmp/installer | ||
| mkdir "${dir}/" | ||
| pushd ${dir} | ||
| cp -t "${dir}" \ | ||
| "${SHARED_DIR}/install-config.yaml" | ||
|
|
||
| ### Create manifests | ||
| echo "Creating manifests..." | ||
| openshift-install --dir="${dir}" create manifests & | ||
|
|
||
| set +e | ||
| wait "$!" | ||
| ret="$?" | ||
| set -e | ||
|
|
||
| if [ $ret -ne 0 ]; then | ||
| cp "${dir}/.openshift_install.log" "${ARTIFACT_DIR}/.openshift_install.log" | ||
| exit "$ret" | ||
| fi | ||
|
|
||
| ### Remove control plane machines | ||
| echo "Removing control plane machines..." | ||
| rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml | ||
|
|
||
| ### Remove compute machinesets (optional) | ||
| echo "Removing compute machinesets..." | ||
| rm -f openshift/99_openshift-cluster-api_worker-machineset-*.yaml | ||
|
|
||
| ### Make control-plane nodes unschedulable | ||
| echo "Making control-plane nodes unschedulable..." | ||
| sed -i "s;mastersSchedulable: true;mastersSchedulable: false;g" manifests/cluster-scheduler-02-config.yml | ||
|
|
||
| ### Create Ignition configs | ||
| echo "Creating Ignition configs..." | ||
| openshift-install --dir="${dir}" create ignition-configs & | ||
|
|
||
| set +e | ||
| wait "$!" | ||
| ret="$?" | ||
| set -e | ||
|
|
||
| cp "${dir}/.openshift_install.log" "${ARTIFACT_DIR}/.openshift_install.log" | ||
|
|
||
| if [ $ret -ne 0 ]; then | ||
| exit "$ret" | ||
| fi | ||
|
|
||
| cp -t "${SHARED_DIR}" \ | ||
| "${dir}/auth/kubeadmin-password" \ | ||
| "${dir}/auth/kubeconfig" \ | ||
| "${dir}/metadata.json" \ | ||
| "${dir}"/*.ign | ||
|
|
||
| # Removed tar of openshift state. Not enough room in SHARED_DIR with terraform state | ||
|
||
|
|
||
| popd | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| ref: | ||
| as: upi-conf-vsphere | ||
| from: upi-installer | ||
| commands: upi-conf-vsphere-commands.sh | ||
| resources: | ||
| requests: | ||
| cpu: 10m | ||
| memory: 100Mi | ||
| documentation: >- | ||
| This step creates the files necessary to deploy a vSphere UPI cluster. | ||
| It requires an existing install-config.yaml to exist in the shared directory with the necessary configuration to deploy a cluster to the vSphere CI project. | ||
| It uses this install-config.yaml to generate the govc.sh, Ignition, metadata.json, and kubeconfig files with the modifications required to support the vSphere UPI workflow. | ||
| These files are copied to the shared directory for future consumption. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| approvers: | ||
| - abhinavdahiya | ||
| - jcpowermac | ||
| - patrickdillon |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be worth teaching ci-operator to pull down timestamped container logs so we don't need to work up one-off manual injection like this. @stevekuznetsov, thoughts?