Maintenance: Change the project.toml to restrict Python 3.10 version

[leandrodamascena]

Summary

The latest version of the Lambda runtime for Python is 3.9 and this project must have the same restriction.
The way project.toml is configured, the package on PyPi adds Python 3.10 as a supported version.

Why is this needed?

Keep the same python versions supported by Lambda Runtime

Which area does this relate to?

Governance, Other

Solution

No response

Acknowledgment

• This request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[jplock]

Locally I’m running Python 3.10 and I install the powertools via pip. When I deploy through SAM I’m using the layer. I don’t think we should limit being able to install this package when running 3.10.

[leandrodamascena]

Hi @jplock, this is really an excellent point for us to debate here. It really works to use Python 3.10 to install via pip or even use the source code of this project, I've done some tests here and it's working "fine". But I think we should pay attention to a few points:

1 - Ok we have Python 3.10 installed and the project working, but we must remember that the highest version allowed by the Lambda Runtime is 3.9. So using local Python 3.10 (even via SAM) to run on the Lambda 3.9 runtime can cause some issues.

2 - Many good things were added in Python 3.10 and of course they are not supported by Python 3.9. Like PEP-0634 and PEP-0604

3 - Now imagine a scenario where a contributor wants to submit a PR for the project, but the contributor has Python 3.10 installed in the local environment and writes code supported only by Python 3.10. When the contributor submits the PR, all pipelines will fail because this project uses matrix to build a version and Python 3.10 isn't supported.
https://github.com/awslabs/aws-lambda-powertools-python/blob/bb1eda742583d58790219942a49a5d2489ea91ea/.github/workflows/python_build.yml#L29

4 - Some examples that work in Python 3.10 and doesn't work in Python 3.9

So based on these facts I think it's protection/better governance to add this restriction to the Python version.

I'd like to hear your point of view to get things I'm missing.

[jplock]

I understand the points but who is our target audience: people who will be submitting changes to the library or people who want to use the library? I think people using the library will be more common and we shouldn’t limit how it can be installed if they happen to use the latest available Python release on their local machines. Maybe we could check with the Lambda runtime team on when 3.10 support is coming? If it’s soon, maybe we skip this.

[leandrodamascena]

I agree with you that people using the library is the most common situation. But in my opinion even those people shouldn't be affected because the purpose of this project is to help projects using AWS Lambda and those projects must use a Python version below 3.10 because Lambda Runtime doesn't support 3.10+.
But I actually don't have enough data and I don't think I'm the right person to define this as I'm not part of the AWS team and I don't have access to all the information. I think we need to hear the views of the maintainers. It's just my opinion and I could be wrong.

Yessss!! Talking to the Lambda runtime team can be awesome as they certainly have a roadmap and can provide more information.

Btw, there is an issue aws/aws-lambda-base-images#31 where people are asking for this feature/support.

[heitorlessa]

We'll look into this for next week's release, as we had some funky expressions with Poetry before. I appreciate everyone's point of view. For correctness, and specially given the number of breaking changes in 3.10+, we should update our support to 3.9 only.

Lambda will take some time to provide support for 3.10. When they do, it'll take us some time to address some difficult typing breakage too.

[peterschutt]

What about those of us running 3.10 in containers via lambda?

Due to the container runtime I could be running 3.11.0b5 as we speak, not that I'd expect this library to support it. But 3.10 has been GA for ~10 months and it is reasonable to expect I wouldn't be the only one running it in a container and depending on this library.

[heitorlessa]

I didn't see that coming. Sometimes I forget about OCI because our tenets track managed runtimes, and that's a blind spot.

This argument is valid and we shouldn't do it since we don't know how many would be impacted; classic Hyrum's Law.

Here's where I'd like help from everyone - how do we best communicate that we don't make guarantees 3.10+ will work?

This applies for any future Python version not yet supported as a Lambda managed runtime.

[peterschutt]

classic Hyrum's Law

one of the best

If I didn't come across this issue today, I wouldn't have known 3.10 was unsupported.

I've looked at awslambdaric as an example, they have pretty much the same classifiers as you, and the same python version specified, but because they use setuptools their classifiers are packaged verbatim:

Whereas poetry seems to assume that the 3.10 classifier is an omission if the python dependency specification includes 3.10. Skimming over the docs, I couldn't see a way to override it either and I'm not sure how I feel about that. Still though, even if the classifier could be rectified, the Requires: Python ... still includes 3.10 in either case:

lambdaric:

you:

I think awslambdaric is a good example though, they pretty prominently state they only support up to 3.9 in their readme, but they don't go so far as to actually restrict 3.10+ and a github search including "awslambdaric" and "3.10" brings up plenty of dockerfiles combining the two.

Similarly, I think it would be a shame for this library to actually put an upper limit on the python version.

Some suggestions:

1. State the version support policy prominently - those tenets are right at the bottom (I'd never actually made it that far down the homepage 😬).
2. Strengthen the wording of the first tenet. IMO it would have been a pretty narrow interpretation of the words "optimise for" to go ahead and exclude python 3.10 from support altogether.
3. You could include a compatibility section of the docs that summarises known version incompatibilities and links to tracking issues.
4. Perhaps a runtime warning on first import if not a supported version - maybe a bit extreme though.
5. If you decide you do want to cap the upper limit in line with the supported runtimes, you could accept that the 3.10 ship has sailed and make the python dependency >=3.6,<=3.10 - that way you wouldn't break anyone now, and wouldn't be forced into 3.11 support when it becomes GA (or whenever poetry decides to add the classifier for you).

You mentioned earlier:

... and specially given the number of breaking changes in 3.10+ ...

Are these breaks tracked anywhere? Searching the issues for "3.10" doesn't come up with much and I've been humming along on 3.10 without any problems until I came across this issue today. Most likely I'm using areas of the lib that aren't affected so it would be good to get a better understanding if you can point me at anything to read.

Cheers:)

[heitorlessa]

That’s super helpful - I’ll read it more carefully this week. For the breaking changes, I meant Python 3.10 and 3.11 breaking typing and some other minor I can’t remember by heart now (e.g., Callable typing). For our breaking changes (v2), the plan is to create an issue and discuss with the community what cuts should we made as part of dropping 3.6 support - I want it to be minimal, and strive for anything major for 3.0 as we aim for modularisation.

…

On Sun, 7 Aug 2022 at 13:26, Peter Schutt ***@***.***> wrote: classic Hyrum's Law one of the best <https://xkcd.com/1172/> If I didn't come across this issue today, I wouldn't have known 3.10 was unsupported. I've looked at awslambdaric as an example, they have pretty much the same classifiers as you, and the same python version specified, but because they use setuptools their classifiers are packaged verbatim: [image: image] <https://user-images.githubusercontent.com/20659309/183286468-5ddbcdda-d066-4218-9d5c-fdc269330df6.png> Whereas poetry seems to assume that the 3.10 classifier is an omission if the python dependency specification includes 3.10. Skimming over the docs, I couldn't see a way to override it either and I'm not sure how I feel about that. Still though, even if the classifier could be rectified, the Requires: Python ... still includes 3.10 in either case: lambdaric: [image: image] <https://user-images.githubusercontent.com/20659309/183287282-680125af-83dd-432b-b8ca-4514538ab68d.png> you: [image: image] <https://user-images.githubusercontent.com/20659309/183287297-d9d7f6ef-29ed-44ea-9f8e-da48e08da8be.png> I think awslambdaric is a good example though, they pretty prominently state they only support up to 3.9 in their readme, but they don't go so far as to actually restrict 3.10+ and a github search including "awslambdaric" and "3.10" brings up plenty of dockerfiles combining the two. Similarly, I think it would be a shame for this library to actually put an upper limit on the python version. Some suggestions: 1. State the version support policy prominently - those tenets are right at the bottom (I'd never actually made it that far down the homepage 😬). 2. Strengthen the wording of the first tenet. IMO it would have been a pretty narrow interpretation of the words "optimise for" to go ahead and exclude python 3.10 from support altogether. 3. You could include a compatibility section of the docs that summarises known version incompatibilities and links to tracking issues. 4. Perhaps a runtime warning on first import if not a supported version - maybe a bit extreme though. 5. If you decide you do want to cap the upper limit in line with the supported runtimes, you could accept that the 3.10 ship has sailed and make the python dependency >=3.6,<=3.10 - that way you wouldn't break anyone now, and wouldn't be forced into 3.11 support when it becomes GA (or whenever poetry decides to add the classifier for you). You mentioned earlier: ... and specially given the number of breaking changes in 3.10+ ... Are these breaks tracked anywhere? Searching the issues for "3.10" <https://github.com/awslabs/aws-lambda-powertools-python/issues?q=is%3Aissue+3.10> doesn't come up with much and I've been humming along on 3.10 without any problems until I came across this issue today. Most likely I'm using areas of the lib that aren't affected so it would be good to get a better understanding if you can point me at anything to read. Cheers:) — Reply to this email directly, view it on GitHub <#1404 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBE4SVTR2LARPAOOBF3VX6MOTANCNFSM55LMNJGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[leandrodamascena]

What about those of us running 3.10 in containers via lambda?

Due to the container runtime I could be running 3.11.0b5 as we speak, not that I'd expect this library to support it. But 3.10 has been GA for ~10 months and it is reasonable to expect I wouldn't be the only one running it in a container and depending on this library.

Ohhh 😮.. I think that's a great argument for not restricting the Python version and I didn't think about this point either. Right now this project could be running on thousands of containers using Python 3.10+ and this could have a big impact.

👏