Correct Way To Create A Package From Protobufs

[inolaole]

About:
We use a repository to generate python files (.py and .pyi) from protobuf definitions. I want to package this repository into a poetry package. I am having trouble finding any relevant documentation outside of this https://github.com/orgs/python-poetry/discussions/10082.

[repo root]
- schemas
   - [org_name]
      - [area (i.e. accounts)] 
         - [sub (v1)]
            - *.proto
         - [sub (v0)]
            - *.proto

which generates .pyi/.py files in the following structure (note that it does not create any init.py files

[repo root]
- generated
   - python
      -  standardproto 
         -  [org_name]
            - [area (i.e. accounts)]
               - [sub (v1)]
                  - *.pyi, *.py
              - [sub (v0)]
                  - *.pyi, *.py

Desire:

1. If possible I would like to learn more about poetry's ability to actually generate the .pyi/.py files using it's protobuf build-backend. I've seen limited documentation on it, and it seems like it might not even exist. If so that's ok, but I'd like to confirm my understanding
2. If poetry can not gen the files that's fine I would just like to know the correct way to define a toml to crawl the generated path, get all of the packages, and then bundle them for distribution via poetry.

[abn]

You can use the include keyword if you want to do this as-is. But unfortunately that won't give you import-able python packages without the __init__.py (you can omit this only for namespace packages). Poetry does not do anything with protobufs.

You can look at https://github.com/python-gnxi/python-gnmi-proto for an example how you could use betterproto to generate python code and then package it with Poetry.

Note that the generation step is done outside of Poetry via the Makefile.

[inolaole]

@abn From what I've pieced together my repo is in fact not generating a correct python package (as is expected from protobufs). I'm looking for the specifics of how you set yours up to be properly distributed.

[inolaole]

As in was this accomplished via virtue of tooling you used, or did you manually create the necessary directory structure such that the output of protobuf would be placed correctly and ready for packaging.

[inolaole]

Thanks @abn your pointers were enough to get me unblocked. I would highly encourage the Poetry docs to be updated with this packages syntax with relevant examples. They were pretty hard to find.

[dimbleby]

I would highly encourage the Poetry docs to be updated

submitting a pull request is the best way to "highly encourage" such things!

[abn]

@inolaole apologies for the delay between responses, my focus has been elsewhere. I will take a look at your approach soon. I have been thinking about all this and might have a better way to approach this. Let me experiment with it over the weekend.

[inolaole]

Alright having solved my problem I will document here for those encountering the same. If you are having issues packaging files generated by the standardproto compiler, the most likely issue you will be facing is that they are missing init.py's at each level of your python packages. In my case

[repo root]
- generated
   - python
      -  standardproto 
         -  [org_name]
            - [area (i.e. accounts)]
               - [sub (v1)]
                  - *.pyi, *.py
              - [sub (v0)]
                  - *.pyi, *.py

In order to properly get these packaged use a script to drop an init.py at each level starting with [org_name]. After you have done that you can specify the following pyproject.toml which will correctly bundle the following. Note that this was run in a special folder that I copy the python files from above into so the structure looks like the following

[repo root]
- poetry
   - [org_name]
     - [area]
       - [sub (v1)]
         - *.py, *.pyi
   - pyproject.toml  

The pyproject.toml will look like

[tool.poetry]
name = "your-python-package"
version = "0.1.0"
description = ""
authors = ["a guy"]
readme = "README.md"
packages = [
    { include = "org_name/**/*.py" },
]

[tool.poetry.dependencies]
python = "^3.11"



[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The net result of which is that when installed this package will drop in <your_python_path>/site-packages/org_name/**/*.py files. Which if you refer to your python protos by their direct namespace should just work.