Skip to content

[Extension] Add azmirror cloud endpoint for installing extensions in air gapped clouds and support config of index url#15128

Merged
fengzhou-msft merged 10 commits intoAzure:devfrom
fengzhou-msft:extension_endpoint
Feb 23, 2021
Merged

[Extension] Add azmirror cloud endpoint for installing extensions in air gapped clouds and support config of index url#15128
fengzhou-msft merged 10 commits intoAzure:devfrom
fengzhou-msft:extension_endpoint

Conversation

@fengzhou-msft
Copy link
Member

@fengzhou-msft fengzhou-msft commented Sep 11, 2020

Description

To support installing extensions in air-gapped clouds, we have set up a pipeline to sync the extension index, wheel packages and extension command tree to a public storage account called azmirror. A Teleport process will be set up to sync everything in azmirror to different storage accounts inside air-gapped clouds.

In order to use those files in each air-gapped cloud with Azure CLI, this PR adds an azmirror_storage_account_resource_id endpoint whose value may look like https://azmirroragc.blob.core.windows.net.
CLI will get extension related files from the endpoint if it is set in the current cloud, otherwise it will use the current default values (for public cloud and sovereign clouds).

This PR also adds config option extension.index_url to support customized extension index url. For example, to use the index.json on github:
az config set extension.index_url=https://raw.githubusercontent.com/Azure/azure-cli-extensions/master/src/index.json

Testing Guide

$ az cloud update --cloud-config "{\"endpoint_azmirror_storage_account_resource_id\":\"https://azureclireleases.blob.core.windows.net/releases\"}"
$ az cloud show  # you should see the value set for azmirrorStorageAccountResourceId
$ az extension list-available -otable --debug  # you should see a request sent to https://azmirror.blob.core.windows.net
$ az config set extension.index_url=https://raw.githubusercontent.com/Azure/azure-cli-extensions/master/src/index.json
$ az extension list-available -otable --debug  # Now you should see a request sent to raw.githubusercontent.com

History Notes

[Extension] Support config of extension index url


This checklist is used to make sure that common guidelines for a pull request are followed.

@yungezz yungezz added the Cloud label Sep 11, 2020
@yungezz yungezz added this to the S175 - For Ignite milestone Sep 11, 2020
@yonzhan
Copy link
Collaborator

yonzhan commented Sep 11, 2020

Extension

Comment on lines +91 to +93
ext_endpoint = cli_ctx.cloud.endpoints.extension_storage_account_resource_id if cli_ctx and \
cli_ctx.cloud.endpoints.has_endpoint_set('extension_storage_account_resource_id') else None
config_index_url = cli_ctx.config.get('extension', 'index_url', None) if cli_ctx else None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can put these 2 URLs in one place?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, isn't the whl location hard-coded in the index?

{
    "downloadUrl": "https://azurecliprod.blob.core.windows.net/cli-extensions/account-0.2.0-py2.py3-none-any.whl",
    "filename": "account-0.2.0-py2.py3-none-any.whl",
    "metadata": {
        "azext.isExperimental": true,

Copy link
Member Author

@fengzhou-msft fengzhou-msft Feb 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The index will be teleported into different storage accounts in air gapped clouds, we do not have permission to modify the whl downloadUrl based on the destination storage accounts after it gets copied in air gapped clouds. So we need to modify the downloadUrl here in azure-cli when we need to download it in an air gapped cloud.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to put one URL in cloud config, one in az config?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The one in cloud config is now a general support for downloading extensions and other tools/files needed in air gapped clouds. It's always a storage account endpoint and controlled by Microsoft.

The one in az config is to support customers that want to use their own customized index.json, which may have their own private extensions and some allowed public extensions. You can use any url of the index file, such as the raw content url on a github branch.

@yungezz yungezz modified the milestones: S175 - For Ignite, Backlog Sep 16, 2020
@yungezz
Copy link
Member

yungezz commented Sep 16, 2020

put this to backlog now given environment setup is under process. When process done, will merge the PR.

synapse_analytics_resource_id=None,
attestation_resource_id=None):
attestation_resource_id=None,
extension_storage_account_resource_id=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extension_pacakges_url?

url = cli_ctx.config.get('extension', 'index_url', None)
if url:
return url
ext_endpoint = cli_ctx.cloud.endpoints.extension_storage_account_resource_id if cli_ctx and \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

config overrides cloud setting

@arrownj arrownj assigned fengzhou-msft and unassigned arrownj Nov 19, 2020
@fengzhou-msft fengzhou-msft changed the title [Extension] Add cloud endpoint for extension and support config of index url [Extension] Add azmirror cloud endpoint for installing extensions in air gapped clouds and support config of index url Feb 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants