azure-eventhub/azure/eventhub/extensions/__init__.py is incompatible with packaging namespaces

[pjachowi]

• Package Name:
  azure-eventhub

• Package Version:
  5.1.0

• Operating System:
  Linux

• Python Version:
  3.8.2

Describe the bug
Existence of emptiness of the file https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/eventhub/azure-eventhub/azure/eventhub/extensions/__init__.py is incompatible with packaging namespaces. It causes problem using this package in Bazel, see dillon-giacoppo/rules_python_external#56

To Reproduce
Steps to reproduce the behavior:

1. Go into temporary directory cd /tmp
2. Install azure_eventhub under dir1 pip install --target=$(pwd)/dir1 azure_eventhub
3. Install azure-eventhub-checkpointstoreblob under dir2 pip install --target=$(pwd)/dir2 azure-eventhub-checkpointstoreblob
4. Create virtual environment virtualenv venv
5. Activate virtual environment source venv/bin/activate
6. Create main.py

cat > main.py
import sys
sys.path.extend(['/tmp/dir1', '/tmp/dir2'])
from azure.eventhub.extensions.checkpointstoreblob import BlobCheckpointStore

7. Execute main.py python ./main.py
8. Result is an error:

Traceback (most recent call last):
  File "main.py", line 3, in <module>
    from azure.eventhub.extensions.checkpointstoreblob import BlobCheckpointStore
ModuleNotFoundError: No module named 'azure.eventhub.extensions.checkpointstoreblob'

Expected behavior
main.py should execute without errors.

Screenshots
N/A

Additional context
The problem emerges because Bazel installs dependencies in separate directories. Current layout, incompatible with packaging namespaces makes it impossible to use in applications build by Bazel, see dillon-giacoppo/rules_python_external#56

[yunhaoling]

hello @pjachowi , thanks for the report, I'm able to reproduce it locally following the steps.

and thanks for the PR, adding __path__ = __import__('pkgutil').extend_path(__path__, __name__) seems to be a viable solution if you want to install package in different locations. I need to do more tests/collect knowledge on packaging to see if it has some impacts that I overlook.

hey @johanste @lmazuel , I remember we've discussed about the how to structure the extension package before, do you have any suggestion on this issue?

[pjachowi]

Hello @yunhaoling, did you have chance to look at the PR? As a side note I've looked into rest of Azure sdk and it looks like __path__ = ... is commonly used in __init__.py.

[pjachowi]

Friendly ping @yunhaoling

[yunhaoling]

hey @pjachowi , apologize for not getting back to you sooner.

Let me give you some more context here:

The empty azure/eventhub/extensions/__init__.py was by-design, it is intended to NOT make azure.eventhub.extensions a namespace package which would give us performance improvement on module searching in python compared to namespace package approach.

We have two plugin packages azure-eventhub-checkpointstoreblob and azure-eventhub-checkpointstoreblob which share the same azure.eventhub.extension namespace. If you check the other two libraries, you would find both azure.eventhub.extension.__init__ files are empty in these two libraries.

However, this approach is causing trouble here -- your use case.

I'm having discussion with our architects on what's the RIGHT approach here, there're majorly two approaches to solve the problem:

1. Making azure.eventhub.extentions a namespace package, just as your PR. But we may need to update the __init__.py in the two plugin libraries as well.
2. Create a separate namespace package library, e.g. azure-eventhub-extensions-nspkg which is shared by azure-eventhub, azure-checkpointstoreblob and azure-checkpointstoreblob-aio. But this approach seems a bit overkill.

The decision would apply to our design of other python sdks which means we have to think about the problem with more care.
Thanks for your understanding and patience.

I'll continue to post updates here and feel free to let us know your thoughts on packaging.

[pjachowi]

Thank you for the explanation.

I've searched Azure/azure-sdk-for-python repository and it seems that __path__ = __import__('pkgutil').extend_path(__path__, __name__) in __init__.py is is commonly used. This query returns 387 results.

What do you think when the changes will be applied?