Skip to content

OCPBUGS-4654: azure: upi: use Image Gallery in ARM templates #6684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 3 commits into from
Jan 18, 2023
Merged

OCPBUGS-4654: azure: upi: use Image Gallery in ARM templates #6684

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2b1f123
OCPBUGS-4654: azure: upi: use Image Gallery in ARM templates
r4f4 Dec 6, 2022
c52afbe
docs: azure/upi: add new required parameters and aarch64 support
r4f4 Dec 8, 2022
12292f0
upi: azure: add parameter for HyperV generation
r4f4 Dec 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions docs/user/azure/install_upi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,8 +195,9 @@ Given the size of the RHCOS VHD, it's not possible to run the deployments with t
We must copy and store it in a storage container instead. To do so, first create a blob storage container and then copy the VHD.

```sh
export OCP_ARCH="x86_64" # or "aarch64"
az storage container create --name vhd --account-name ${CLUSTER_NAME}sa
export VHD_URL=$(openshift-install coreos print-stream-json | jq -r '.architectures.x86_64."rhel-coreos-extensions"."azure-disk".url')
export VHD_URL=$(openshift-install coreos print-stream-json | jq -r --arg arch "$OCP_ARCH" '.architectures[$arch]."rhel-coreos-extensions"."azure-disk".url')
az storage blob copy start --account-name ${CLUSTER_NAME}sa --account-key $ACCOUNT_KEY --destination-blob "rhcos.vhd" --destination-container vhd --source-uri "$VHD_URL"
```

Expand Down Expand Up @@ -288,11 +289,15 @@ Create the deployment using the `az` client:

```sh
export VHD_BLOB_URL=`az storage blob url --account-name ${CLUSTER_NAME}sa --account-key $ACCOUNT_KEY -c vhd -n "rhcos.vhd" -o tsv`
export STORAGE_ACCOUNT_ID=`az storage account show -g ${RESOURCE_GROUP} --name ${CLUSTER_NAME}sa --query id -o tsv`
export AZ_ARCH=`echo $OCP_ARCH | sed 's/x86_64/x64/;s/aarch64/Arm64/'`

az deployment group create -g $RESOURCE_GROUP \
--template-file "02_storage.json" \
--parameters vhdBlobURL="$VHD_BLOB_URL" \
--parameters baseName="$INFRA_ID"
--parameters baseName="$INFRA_ID" \
--parameters storageAccount="${CLUSTER_NAME}sa" \
--parameters architecture="$AZ_ARCH"
```

## Deploy the load balancers
Expand Down
170 changes: 141 additions & 29 deletions upi/azure/02_storage.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,41 +1,153 @@
{
"$schema" : "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion" : "1.0.0.0",
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
"description" : "Base name to be used in resource names (usually the cluster's Infra ID)"
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"architecture": {
"type": "string",
"metadata": {
"description": "The architecture of the Virtual Machines"
},
"defaultValue": "x64",
"allowedValues": [
"Arm64",
"x64"
]
},
"baseName": {
"type": "string",
"minLength": 1,
"metadata": {
"description": "Base name to be used in resource names (usually the cluster's Infra ID)"
}
},
"storageAccount": {
"type": "string",
"metadata": {
"description": "The Storage Account name"
}
},
"vhdBlobURL" : {
"type" : "string",
"metadata" : {
"description" : "URL pointing to the blob where the VHD to be used to create master and worker machines is located"
"vhdBlobURL": {
"type": "string",
"metadata": {
"description": "URL pointing to the blob where the VHD to be used to create master and worker machines is located"
}
}
},
"variables" : {
"location" : "[resourceGroup().location]",
"imageName" : "[concat(parameters('baseName'), '-image')]"
"variables": {
"location": "[resourceGroup().location]",
"galleryName": "[concat('gallery_', replace(parameters('baseName'), '-', '_'))]",
Copy link
Contributor

@MayXuQQ MayXuQQ Dec 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about concat 'architecture' to the galleryName? since 'arm64' and 'amd64' workers can be added to a cluster. @r4f4

Copy link
Contributor Author

@r4f4 r4f4 Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do that, then we need to pass the architecture parameter to the VM templates (bootstrap, masters, and workers) since they reference the gallery by name.

Copy link
Contributor

@Prashanth684 Prashanth684 Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would not do this since in the future when we support installing a multi-arch compute cluster through installer, we would be creating arm images under the same gallery. we don't want to be creating a separate gallery for arm instances/x86 instances. maybe the image inside the gallery could be denoted with -arch suffix.

Copy link
Contributor Author

@r4f4 r4f4 Dec 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another problem with changing the gallery name is that we also need to change https://github.com/openshift/installer/blob/master/pkg/asset/machines/azure/machines.go#L120 otherwise we break scaling with MAO.

"imageName": "[parameters('baseName')]",
"imageNameGen2": "[concat(parameters('baseName'), '-gen2')]",
"imageRelease": "1.0.0"
},
"resources" : [
"resources": [
{
"apiVersion" : "2018-06-01",
"type": "Microsoft.Compute/images",
"name": "[variables('imageName')]",
"location" : "[variables('location')]",
"properties": {
"storageProfile": {
"osDisk": {
"osType": "Linux",
"apiVersion": "2021-10-01",
"type": "Microsoft.Compute/galleries",
"name": "[variables('galleryName')]",
"location": "[variables('location')]",
"resources": [
{
"apiVersion": "2021-10-01",
"type": "images",
"name": "[variables('imageName')]",
"location": "[variables('location')]",
"dependsOn": [
"[variables('galleryName')]"
],
"properties": {
"architecture": "[parameters('architecture')]",
"hyperVGeneration": "V1",
"identifier": {
"offer": "rhcos",
"publisher": "RedHat",
"sku": "basic"
},
"osState": "Generalized",
"blobUri": "[parameters('vhdBlobURL')]",
"storageAccountType": "Standard_LRS"
}
"osType": "Linux"
},
"resources": [
{
"apiVersion": "2021-10-01",
"type": "versions",
"name": "[variables('imageRelease')]",
"location": "[variables('location')]",
"dependsOn": [
"[variables('imageName')]"
],
"properties": {
"publishingProfile": {
"storageAccountType": "Standard_LRS",
"targetRegions": [
{
"name": "[variables('location')]",
"regionalReplicaCount": "1"
}
]
},
"storageProfile": {
"osDiskImage": {
"source": {
"id": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccount'))]",
"uri": "[parameters('vhdBlobURL')]"
}
}
}
}
}
]
},
{
"apiVersion": "2021-10-01",
"type": "images",
"name": "[variables('imageNameGen2')]",
"location": "[variables('location')]",
"dependsOn": [
"[variables('galleryName')]"
],
"properties": {
"architecture": "[parameters('architecture')]",
"hyperVGeneration": "V2",
"identifier": {
"offer": "rhcos-gen2",
"publisher": "RedHat-gen2",
"sku": "gen2"
},
Copy link
Contributor

@MayXuQQ MayXuQQ Dec 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just use the market-place image?

Copy link
Contributor Author

@r4f4 r4f4 Dec 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would that work for aarch64/Arm64?

$: az vm image list --all --offer rh-ocp-worker --publisher redhat | jq '.[] | .architecture' | sort | uniq
"x64"

Copy link
Contributor

@Prashanth684 Prashanth684 Dec 14, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

marketplace images are not available for arm64, only for x86

"osState": "Generalized",
"osType": "Linux"
},
"resources": [
{
"apiVersion": "2021-10-01",
"type": "versions",
"name": "[variables('imageRelease')]",
"location": "[variables('location')]",
"dependsOn": [
"[variables('imageNameGen2')]"
],
"properties": {
"publishingProfile": {
"storageAccountType": "Standard_LRS",
"targetRegions": [
{
"name": "[variables('location')]",
"regionalReplicaCount": "1"
}
]
},
"storageProfile": {
"osDiskImage": {
"source": {
"id": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccount'))]",
"uri": "[parameters('vhdBlobURL')]"
}
}
}
}
}
]
}
}
]
}
]
}
}
16 changes: 14 additions & 2 deletions upi/azure/04_bootstrap.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,17 @@
"metadata" : {
"description" : "The size of the Bootstrap Virtual Machine"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
"description": "VM generation image to use"
},
"defaultValue": "V2",
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
Expand All @@ -50,7 +61,8 @@
"identityName" : "[concat(parameters('baseName'), '-identity')]",
"vmName" : "[concat(parameters('baseName'), '-bootstrap')]",
"nicName" : "[concat(variables('vmName'), '-nic')]",
"imageName" : "[concat(parameters('baseName'), '-image')]",
"galleryName": "[concat('gallery_', replace(parameters('baseName'), '-', '_'))]",
"imageName" : "[concat(parameters('baseName'), if(equals(parameters('hyperVGen'), 'V2'), '-gen2', ''))]",
"clusterNsgName" : "[concat(if(not(empty(parameters('vnetBaseName'))), parameters('vnetBaseName'), parameters('baseName')), '-nsg')]",
"sshPublicIpAddressName" : "[concat(variables('vmName'), '-ssh-pip')]"
},
Expand Down Expand Up @@ -132,7 +144,7 @@
},
"storageProfile" : {
"imageReference": {
"id": "[resourceId('Microsoft.Compute/images', variables('imageName'))]"
"id": "[resourceId('Microsoft.Compute/galleries/images', variables('galleryName'), variables('imageName'))]"
},
"osDisk" : {
"name": "[concat(variables('vmName'),'_OSDisk')]",
Expand Down
16 changes: 14 additions & 2 deletions upi/azure/05_masters.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,17 @@
"metadata" : {
"description" : "Size of the Master VM OS disk, in GB"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
"description": "VM generation image to use"
},
"defaultValue": "V2",
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
Expand All @@ -70,7 +81,8 @@
"internalLoadBalancerName" : "[concat(parameters('baseName'), '-internal-lb')]",
"sshKeyPath" : "/home/core/.ssh/authorized_keys",
"identityName" : "[concat(parameters('baseName'), '-identity')]",
"imageName" : "[concat(parameters('baseName'), '-image')]",
"galleryName": "[concat('gallery_', replace(parameters('baseName'), '-', '_'))]",
"imageName" : "[concat(parameters('baseName'), if(equals(parameters('hyperVGen'), 'V2'), '-gen2', ''))]",
"copy" : [
{
"name" : "vmNames",
Expand Down Expand Up @@ -144,7 +156,7 @@
},
"storageProfile" : {
"imageReference": {
"id": "[resourceId('Microsoft.Compute/images', variables('imageName'))]"
"id": "[resourceId('Microsoft.Compute/galleries/images', variables('galleryName'), variables('imageName'))]"
},
"osDisk" : {
"name": "[concat(variables('vmNames')[copyIndex()], '_OSDisk')]",
Expand Down
16 changes: 14 additions & 2 deletions upi/azure/06_workers.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,17 @@
"metadata" : {
"description" : "The size of the each Node Virtual Machine"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
"description": "VM generation image to use"
},
"defaultValue": "V2",
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
Expand All @@ -55,7 +66,8 @@
"infraLoadBalancerName" : "[parameters('baseName')]",
"sshKeyPath" : "/home/capi/.ssh/authorized_keys",
"identityName" : "[concat(parameters('baseName'), '-identity')]",
"imageName" : "[concat(parameters('baseName'), '-image')]",
"galleryName": "[concat('gallery_', replace(parameters('baseName'), '-', '_'))]",
"imageName" : "[concat(parameters('baseName'), if(equals(parameters('hyperVGen'), 'V2'), '-gen2', ''))]",
"copy" : [
{
"name" : "vmNames",
Expand Down Expand Up @@ -130,7 +142,7 @@
},
"storageProfile" : {
"imageReference": {
"id": "[resourceId('Microsoft.Compute/images', variables('imageName'))]"
"id": "[resourceId('Microsoft.Compute/galleries/images', variables('galleryName'), variables('imageName'))]"
},
"osDisk" : {
"name": "[concat(variables('vmNames')[copyIndex()],'_OSDisk')]",
Expand Down