Emit one Python module per Zserio package #437
Replies: 12 comments
-
Thank you to bring this topic up. Your idea one Python module per Zserio package is very good and I guess that Python users will be very happy with this very intuitive solution. We have even tried to implement this during the first Python generator design. Unfortunately, we have run to difficulties for which we were not able to find out any easy solution. It might be that we overlooked something, so it is worth to discuss. Difficulties mean the symbol dependencies. Zserio compiler naturally allows dependencies which are defined after reference. Example:
Python does not allow such dependencies, each referenced symbol must be defined before it is used. We were not able to find out in Python something like forward declarations in C++. There are some workarounds available (like using class decorators) but this will be very difficult to implement in generated code in generic way. We might think to restrict language not to allow such dependencies. This could be probably less pain for constants. But for normal types, this is a basic stone of the zserio language which will be required by users. Quite often, we have a conflict between supported languages. Although, some solution makes perfect sense for one language (Python), it does not make any sense for another (Java). So if we restrict zserio compiler core to satisfy needs of one language, the users of different language will be very unpleasantly surprised. Regarding of the problem with imports, we understand that this is a pain and error prone. Because of that each package has generated
|
Beta Was this translation helpful? Give feedback.
-
Thanks a lot for taking the time to write this detailed explanation @mikir ! I didn't know that zserio allows such declarations. The best fix that I can think of would be to move the symbol imports from the
Zserio would need to generate:
The contents of import .a, .b, .item, .main
A = a.A
B = b.B
Item = item.Item
Main = main.Main That would solve the issue I think. Personally, I would even support prefixing the symbol-modules with an underscore to denote them as private and discourage deeply nested import statements. |
Beta Was this translation helpful? Give feedback.
-
Yes, I agree, Difficulties were called 'Circular Import Problem'. And we were not able to solve this in generic way. But again, we could forgot for some easy solution. The following explains the problem. Consider the following python project layout:
Source
Source
Source
Source
Then trying to import module
This error can be solved by removing full module paths in
But this solution would break completely the clashing resolution in the generated Python sources. Therefore, we have introduced |
Beta Was this translation helpful? Give feedback.
-
One of the problems in this case is Edit: Please see next comment.
|
Beta Was this translation helpful? Give feedback.
-
An easy solution might be to hide the original import structure completely behind an _impl (or similar) package:
|
Beta Was this translation helpful? Give feedback.
-
@josephbirkner In the meantime please consider to use the generated
Regardless of what it generates underneath, you can just import the top level
Also please note that we generate the code. It's always quite easy to fix one particular example by hand, but it wasn't too easy for us to solve it in general. In the time when we implemented pyhton support, we weren't able to solve issues with If we decide to change the way we do it, we must be absolutely sure that we are able to solve all the use-cases generally, so we must consider it very carefully. |
Beta Was this translation helpful? Give feedback.
-
Sounds very good. Sorry for my trail of edits, I had to go through quite a thought process myself😅 The |
Beta Was this translation helpful? Give feedback.
-
Don't worry about edits. :-) It's not so easy and obvious. It is good to discuss. The more heads, the more knowledge. It might be that we find out something better in the future. |
Beta Was this translation helpful? Give feedback.
-
Another problem with emitting one Python module per Zserio package is cyclic imports which are allowed in Zserio. |
Beta Was this translation helpful? Give feedback.
-
Also note that the approach with separate module for each Zserio entity works without need of cyclic dependencies in target language because Zserio has only limited support of recursion - only "direct" recursion is allowed. Thus when e.g. |
Beta Was this translation helpful? Give feedback.
-
That makes sense. I am very ok with the current solution, so you may close this issue at your convenience 😃 |
Beta Was this translation helpful? Give feedback.
-
We have created a corresponded design note to put together what we have learned in this issue. |
Beta Was this translation helpful? Give feedback.
-
Please consider changing the python-module-per-zserio-symbol behavior. It would be really nice if I could just write
rather than
Aside from the PEP-8 clash, the latter just feels very redundant, and occasionally I forget that I have to import the nested Symbol and obviously get very weird errors. It would also bring the imports in Python more in line with the imports in Zserio. Thanks!
I think this issue is also relevant for other extensions (e.g. C++, see #274), but the solution might be more straightforward for Python.
Edit for visitors:
Use the generated
api
module as a workaround:Beta Was this translation helpful? Give feedback.
All reactions