[Extension] Allow to load extensions in the system path via packages #12367

bluca · 2020-02-26T22:33:48Z

On Linux, check if there are extension in the system Python
library path (eg: /usr/lib/python3/dist-packages/azure-cli-extensions).

This allows extensions to be installed via system packages in
dist-packages, and registered by simply creating an extension directory
and symlinking the egg-info and the module.

EG:

$ ls -l /usr/lib/python3/dist-packages/azure-cli-extensions/azure-devops/
total 0
lrwxrwxrwx 1 root root 18 Feb 23 12:31 azext_devops -> ../../azext_devops
lrwxrwxrwx 1 root root 34 Feb 23 12:31 azure_devops-0.17.0.egg-info -> ../../azure_devops-0.17.0.egg-info
$ az extension list
[
{
"extensionType": "whl",
"name": "azure-devops",
"version": "0.17.0"
}
]

Locally installed extensions still have higher priority if available.

On Linux, check if there are extension in the system Python library path (eg: /usr/lib/python3/dist-packages/azure-cli-extensions). This allows extensions to be installed via system packages in dist-packages, and registered by simply creating an extension directory and symlinking the egg-info and the module. EG: $ ls -l /usr/lib/python3/dist-packages/azure-cli-extensions/azure-devops/ total 0 lrwxrwxrwx 1 root root 18 Feb 23 12:31 azext_devops -> ../../azext_devops lrwxrwxrwx 1 root root 34 Feb 23 12:31 azure_devops-0.17.0.egg-info -> ../../azure_devops-0.17.0.egg-info $ az extension list [ { "extensionType": "whl", "name": "azure-devops", "version": "0.17.0" } ] Locally installed extensions still have higher priority if available.

yungezz · 2020-02-27T00:44:56Z

HI @fengzhou-msft could you pls help to have a look? thanks.

haroldrandom · 2020-03-23T03:37:33Z

Why Linux only? As far as I know, windows should also work for this.

haroldrandom · 2020-03-23T03:51:23Z

src/azure-cli-core/azure/cli/core/extension/__init__.py


 from azure.cli.core._config import GLOBAL_CONFIG_DIR, ENV_VAR_PREFIX
-
+from distutils.sysconfig import get_python_lib


What if there is extension containing C code (non-pure Python), will the get_python_lib() work?

On my Debian box cpython libraries are in dist-packages too, so I'm not expecting issues:

>>> from distutils.sysconfig import get_python_lib >>> get_python_lib() '/usr/lib/python3/dist-packages'

$ find /usr/lib/python3/dist-packages/ -name *.so -type f | wc -l 241

Also, what we are after is the azure-cli-extensions directory where we store the symlinks - the symlinks themselves can point to anywhere

bluca · 2020-03-23T08:52:36Z

Why Linux only? As far as I know, windows should also work for this.

Because I can only test it on Linux :-) If someone can confirm it works on Windows too, and we can establish a canonical installation path, then we can add it in the future. OSX too.

But I'm not sure it's too interesting for other OSes? The use case is Linux distro packages, not sure Win/OSX match that model? But I'm really not familiar with deployments in those OSes, so I might very well be wrong.

fengzhou-msft · 2020-03-27T09:35:03Z

@bluca While this PR adds a useful feature to use extensions in the system path, I plan to make the system directory configurable just like the user extension directory. And it's also useful to add an option in az extension add to install extensions in the system directory. I will work on it in another PR.

bluca · 2020-03-27T09:37:18Z

@bluca While this PR adds a useful feature to use extensions in the system path, I plan to make the system directory configurable just like the user extension directory. And it's also useful to add an option in az extension add to install extensions in the system directory. I will work on it in another PR.

That's fine, as long as the default works as-is without any additional configuration - otherwise it's not useful

bluca · 2020-03-27T09:39:06Z

@bluca While this PR adds a useful feature to use extensions in the system path, I plan to make the system directory configurable just like the user extension directory. And it's also useful to add an option in az extension add to install extensions in the system directory. I will work on it in another PR.

That's fine, as long as the default works as-is without any additional configuration - otherwise it's not useful

@fengzhou-msft can we start by merging this and then doing additional work on top?

fengzhou-msft · 2020-03-27T09:40:21Z

@bluca While this PR adds a useful feature to use extensions in the system path, I plan to make the system directory configurable just like the user extension directory. And it's also useful to add an option in az extension add to install extensions in the system directory. I will work on it in another PR.

That's fine, as long as the default works as-is without any additional configuration - otherwise it's not useful

@fengzhou-msft can we start by merging this and then doing additional work on top?

Yes. That's the plan.

bluca · 2020-03-27T09:46:02Z

Great, thank you!

fengzhou-msft · 2020-03-27T14:03:26Z

@bluca on Ubuntu, our package uses a wrapped python instead of the system python, and the python_lib path is /opt/az/lib/python3.6/site-packages, since you mentioned using symlinks so I think it would also work for you, but not as convenient? On CentOS/RHEL, we use the system python.

bluca · 2020-03-27T14:08:23Z

I don't use your packages because of that. I've uploaded azure-cli to Debian and Ubuntu using system's python and dependencies.

https://packages.debian.org/sid/azure-cli
https://packages.ubuntu.com/focal/azure-cli

bluca · 2020-03-27T14:12:52Z

Thanks!

jiasli · 2021-11-17T03:55:35Z

@bluca, after we supported Python 3.10 (#20195), distutils.sysconfig including its function get_python_lib has been deprecated in Python 3.10:

>>> from distutils.sysconfig import get_python_lib
<stdin>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<stdin>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead

According to https://bugs.python.org/issue41282#msg393021, it is not recommended/possible to get /usr/lib/python3/dist-packages with the new sysconfig.get_path:

>>> import distutils.sysconfig
>>> distutils.sysconfig.get_python_lib()
'/usr/lib/python3/dist-packages'
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/lib/python3.8', 'platstdlib': '/usr/lib/python3.8', 'purelib': '/usr/lib/python3.8/site-packages', 'platlib': '/usr/lib/python3.8/site-packages', 'include': '/usr/include/python3.8', 'platinclude': '/usr/include/python3.8', 'scripts': '/usr/bin', 'data': '/usr'}

distutils.sysconfig in Debian seems to be specially customized:

cat /usr/lib/python3.8/distutils/sysconfig.py

def get_python_lib(plat_specific=0, standard_lib=0, prefix=None):
    ...
            return os.path.join(prefix, "lib", "python3", "dist-packages")

Questions:

Do you have any idea for how we should fix this?
What is the Debian's solution to the distutils.sysconfig deprecation?

bluca · 2021-11-17T11:47:27Z

I think it's still being discussed. As a temporary workaround you could do something like sysconfig.get_paths()['platlib'].replace("site", "dist") but it's just a warning for now, so I'd suggest to wait

jiasli · 2021-11-17T12:05:20Z

It doesn't seem distutils.sysconfig.get_python_lib() and sysconfig.get_paths()['platlib'].replace("site", "dist") are the same:

>>> import distutils.sysconfig
>>> import sysconfig
>>> distutils.sysconfig.get_python_lib()
'/usr/lib/python3/dist-packages'
>>> sysconfig.get_paths()['platlib'].replace("site", "dist")
'/usr/lib/python3.8/dist-packages'

but it's just a warning for now, so I'd suggest to wait

Now it is flooding our CI with DeprecationWarning. I would prefer to fix it, even with a hardcode/workaround.

bluca · 2021-11-17T12:10:55Z

Right, it doesn't seem like Python added a way to get that path then - you can just hard-code it until they fix it, if you want to get rid of the warnings I guess

jiasli · 2021-11-17T12:27:52Z

Thanks @bluca, this is helpful information. Could you help create a PR? It is difficult for me to test it. Appreciate it!

bluca · 2021-11-17T15:41:47Z

Sure, this works for me: #20391

bluca requested review from Juliehzl, arrownj, fengzhou-msft, haroldrandom, jiasli and qianwens as code owners February 26, 2020 22:33

bluca changed the title ~~Allow to install extensions in the system path via packages~~ [Extensions] Allow to install extensions in the system path via packages Feb 26, 2020

bluca force-pushed the system_extensions branch from 8724142 to 1001e0f Compare February 26, 2020 22:58

yungezz assigned fengzhou-msft Feb 27, 2020

yungezz added this to the S166 milestone Feb 27, 2020

fengzhou-msft modified the milestones: S166, S167 Mar 11, 2020

haroldrandom reviewed Mar 23, 2020

View reviewed changes

fengzhou-msft approved these changes Mar 27, 2020

View reviewed changes

fengzhou-msft merged commit 4dfb0f8 into Azure:dev Mar 27, 2020

bluca deleted the system_extensions branch March 27, 2020 14:12

fengzhou-msft changed the title ~~[Extensions] Allow to install extensions in the system path via packages~~ [Extension] Allow to install extensions in the system path via packages Mar 30, 2020

fengzhou-msft changed the title ~~[Extension] Allow to install extensions in the system path via packages~~ [Extension] Allow to load extensions in the system path via packages Mar 30, 2020

This was referenced May 1, 2020

[Extension] Support installing extensions in a system directory #12856

Merged

[Extension] support .egg-info to store wheel type extension metadata #13286

Merged


		from azure.cli.core._config import GLOBAL_CONFIG_DIR, ENV_VAR_PREFIX

		from distutils.sysconfig import get_python_lib

[Extension] Allow to load extensions in the system path via packages #12367

[Extension] Allow to load extensions in the system path via packages #12367

Uh oh!

Conversation

bluca commented Feb 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yungezz commented Feb 27, 2020

Uh oh!

haroldrandom commented Mar 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haroldrandom Mar 23, 2020

Choose a reason for hiding this comment

Uh oh!

bluca Mar 23, 2020

Choose a reason for hiding this comment

Uh oh!

bluca commented Mar 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fengzhou-msft commented Mar 27, 2020

Uh oh!

bluca commented Mar 27, 2020

Uh oh!

bluca commented Mar 27, 2020

Uh oh!

fengzhou-msft commented Mar 27, 2020

Uh oh!

bluca commented Mar 27, 2020

Uh oh!

fengzhou-msft commented Mar 27, 2020

Uh oh!

bluca commented Mar 27, 2020

Uh oh!

bluca commented Mar 27, 2020

Uh oh!

jiasli commented Nov 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bluca commented Nov 17, 2021

Uh oh!

jiasli commented Nov 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bluca commented Nov 17, 2021

Uh oh!

jiasli commented Nov 17, 2021

Uh oh!

bluca commented Nov 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bluca commented Feb 26, 2020 •

edited

Loading

haroldrandom commented Mar 23, 2020 •

edited

Loading

bluca commented Mar 23, 2020 •

edited

Loading

jiasli commented Nov 17, 2021 •

edited

Loading

jiasli commented Nov 17, 2021 •

edited

Loading