Django Google Summer of Code

[adamchainz]

Just a heads-up to other maintainers like @sobolevn and @intgr : django-stubs, and type hints in general, are again on the list of possible GSoC projects for Django this year.

Carlton summarized the current status on this post, with a link to the wiki page with draft project descriptions: https://groups.google.com/g/django-developers/c/0lelyphIw8k/m/Hz2b6OD_BwAJ

Doesn't mean any students will apply or be accepted to work on this, but if they do, it would be good to have some mentors!

[flaeppe]

Just wanted to point out some examples I've seen in the wild where there's been work in (quite large) codebases to contribute type hints "afterwards":

• Celery type hints tracking issue (ongoing)
• Kombu type hints tracking issue (ongoing)
• uvicorn type hints tracking issue (complete)

(Edit: By just browsing the above links, one can find links to other projects attempting the same approach)

It just so happens that it's the same person taking charge (Kludex).

Thinking if one wants some reference of how it can be done, or just take inspiration from 1 approach of how to do it. And also where it's been successful, at least for one project.

There's also an example of tooling that has been used here trying to auto populate missing annotations: #909. Think I've seen scripts like that in other projects too, but can't recall where.

[sobolevn]

I am not able to participate in GSoC anymore, I have the wrong nationality for them :(

[intgr]

work in (quite large) codebases to contribute type hints "afterwards"

Are you suggesting that the type hints from django-stubs should be pushed upstream to Django proper? Last time this was discussed on Django mailing lists, their stance was that the typing ecosystem in Python is still too immature and they don't want upstream hints.

Without a change in that policy, this project would be dead in the water.

---

However, there are a few "pie in the sky" projects that would be very helpful for django-stubs and I can consider mentoring:

• Integrate mypy_primer with our CI, so that every django-stubs PR will get tested with a bunch of open source Django projects, to gain better visibility into how much breakage we are causing.

• The mypy plugin should be "hygienic" with regards to Django code being checked. It should not directly import project code or Django into the mypy runtime, so it doesn't get entangled with the code it's checking. Instead it could for example invoke subprocesses to discover Django internals.

  The current implementation is problematic for many reasons. For instance mypy shouldn't crash with traceback when there's a syntax error in Django settings. And when using mypy daemon dmypy, it should be capable of re-discovering changed models files, instead of using obsolete state or randomly crashing, as it does now.

EDIT: I decided not to apply for these, I don't have enough understanding of either to be a competent mentor.

[ngnpope]

Without a change in that policy, this project would be dead in the water.

There's talk of an exploratory proof of concept for doing this in core Django as part of GSoC 2023...

[carltongibson]

There's talk of an exploratory proof of concept for doing this in core Django as part of GSoC 2023...

I'm looking for input on what would be feasible there. What is the state of play in 2023 (recalling that anything we add now will need to be ≈stable for 5 versions of Python before it EOLs with the then-old-LTS).

There's still lots of churn, but what's stable? What can we do with Protocols and such? Would adding (even a beginning) of a django.types module help?

Typing in Python is evolving, and becoming more popular… can we begin? (Maybe the answer is still No — but perhaps we can get closer to what a Yes would need to look like…)

As the idea says "exploratory" — but I don't think we need to just say No forever because that was the answer three versions of Python ago.

• Adam has been using and written a lot on typing.
• Simon (Charette) explicitly raised the possibility of typing internal parts of the ORM in his DjangoCon US keynote in October 2022.

Both those folks are current Steering Council members — so it's not in any way the case that nothing would be considered here.

But what?

I hope that makes sense. All input welcomed!

[adamchainz]

Integrate mypy_primer with our CI, so that every django-stubs PR will get tested with a bunch of open source Django projects, to gain better visibility into how much breakage we are causing.

💯 great idea.

• The mypy plugin should be "hygienic" with regards to Django code being checked. It should not directly import project code or Django into the mypy runtime, so it doesn't get entangled
  with the code it's checking. Instead it could for example invoke subprocesses to discover Django internals.

Probably a bit of a brain twister but would be nice to see. I think it would also make it a bit easier to separate out what needs discovering and open the door for other type checkers.

There's talk of an exploratory proof of concept for doing this in core Django as part of GSoC 2023...

My feeling now is that we could start by typing Django internally, running a fairly lax Mypy setup. We could target certain areas at a time, like some ORM internals as suggested by Simon. But Django wouldn't contain a py.typed file so the type hints wouldn't (technically) be exposed to users.

For users, django-stubs could remain the solution. For contributors, we'd gradually add types to one area at a time. Eventually they meet in the middle?

[Kludex]

Just wanted to point out some examples I've seen in the wild where there's been work in (quite large) codebases to contribute type hints "afterwards":

• Celery type hints tracking issue (ongoing)
• Kombu type hints tracking issue (ongoing)
• uvicorn type hints tracking issue (complete)

(Edit: By just browsing the above links, one can find links to other projects attempting the same approach)

It just so happens that it's the same person taking charge (Kludex).

Thinking if one wants some reference of how it can be done, or just take inspiration from 1 approach of how to do it. And also where it's been successful, at least for one project.

There's also an example of tooling that has been used here trying to auto populate missing annotations: #909. Think I've seen scripts like that in other projects too, but can't recall where.

Not sure if helps on something, but I've created this small script to create that table on the issues: https://github.com/Kludex/annotation-plan 😅

[sobolevn]

🎉 https://code.djangoproject.com/wiki/SummerOfCode2023#TypedDjango

[RyanWalker277]

Hey everyone! So I was looking for a new issue to work upon when I stumbled across this discussion. I don't know if this is the right place for me to talk, but being a GSOC2023 aspirant I couldn't resist myself. I was a django enthusiast during my early development days, but it was during my scavageing period for GSOC that I discovered django-stubs and python type checking. I found the whole idea of type-checking interesting. I am now trying to learn more about it and contribute to this project. I did some research on the idea of integrating mypy primer into the CI and really liked it. I wanted to work on that but as @intgr said in #1362, there are no current plans for this, I started looking for something else. But seeing the idea recognised as a potential GSOC idea in this discussion I couldn't resist myself from speaking out here.I would like to work on this and wanted to ask if I could include this in my proposal alongside the current mentioned idea ( Strict-mode support for django-stubs ). I apologise if this thread wasn't made for participant discussions.

[sobolevn]

@RyanWalker277 awesome that you are interested in helping us! 🎉

I think we can have the informal chat here, but the final application needs to go through Google's official channel.

I think that mypy_primer project is much harder. Because there are known problems with installing plugins. And since all django-stubs users are implied to use our plugin, we will have to figure this out. But, since mypy_primer is not owned or exclusively used by us, it will be quite hard to figure out: not to break existing usages in mypy and typeshed, etc.

So, I think that strict mode is a better project :)

[intgr]

I did some research on the idea of integrating mypy primer into the CI and really liked it. I wanted to work on that but as @intgr said in #1362, there are no current plans for this

Sorry, I forgot to post an update here. Yeah, I considered submitting django-stubs mypy_primer integration for GSoC, but I decided not to. I myself have no experience with it and I would have to learn it from scratch to be a good mentor. Same with the hygienic plugin redesign.

Unless @hauntsaninja is interested to apply as a mentor for that project.

[intgr]

For clarity, I have separated this hygienic plugin discussion to another issue:

• Hygienic mypy plugin: don't import Django into the mypy process #1384

[RyanWalker277]

I think that mypy_primer project is much harder. Because there are known problems with installing plugins. And since all django-stubs users are implied to use our plugin, we will have to figure this out. But, since mypy_primer is not owned or exclusively used by us, it will be quite hard to figure out: not to break existing usages in mypy and typeshed, etc.

So, I think that strict mode is a better project :)

Sorry, I forgot to post an update here. Yeah, I considered submitting django-stubs mypy_primer integration for GSoC, but I > decided not to. I myself have no experience with it and I would have to learn it from scratch to be a good mentor. Same with the > hygienic plugin redesign.

Thanks for your response! I appreciate the insights on the technical challenges of integrating mypy_primer into the CI for django-stubs. I now understand that it's a complex issue that requires careful consideration and planning.

I'll focus on the strict-mode support for django-stubs project for my GSOC2023 proposal. However, I'm still very interested in learning more about the technical details of mypy_primer and will be exploring it as a side project.

If you have any resources or suggestions on where I can start, I'd love to hear them! Thanks again for your help and guidance.

[sobolevn]

@RyanWalker277 you are welcome! Feel free to ask me any questions about both projects. I have some experience with using and integrating mypy_primer :)

[RyanWalker277]

Hey @sobolevn @intgr ! I've completed my proposal and it's ready for review. I have taken up the strict mode idea as a base goal and added the mypy_primer idea as a stretch goal. It would mean a lot to me if you and others could take a look at it and provide some feedback :)

https://docs.google.com/document/d/14wvLqdHXz117H3_mUvZ_rI4efoaWulLO1X79QxZD6ek/edit?usp=sharing

[sobolevn]

@RyanWalker277 nice! Since you don't have comments enabled, I will share my suggestions here if you don't mind:

Mypy would not flag this as an error because List is a subtype of Sequence and therefore is compatible with any other sequence type

This is not quite true, because list is invariant.

The solution to this is pretty straightforward. Adding proper type parameters to all the declarations makes it pass mypy’s strict mode check.

In your solution you are just adding Any. I agree that in some cases this is the correct thing to do, but I think that adding a single sentence about one of the subgoals would be benefitial: "inferring correct type arguments in different places in code".

Other than that - it is a great proposal! Looking forward to working together :)

Thanks to @carltongibson and @adamchainz for making this project possible. 🤝

[RyanWalker277]

Since you don't have comments enabled, I will share my suggestions here if you don't mind

Really sorry for the inconvinience, I wasn't aware that comments aren't enabled by default 🥲

This is not quite true, because list is invariant.

Thanks for pointing it out. I have changed it accordingly, I think it would be fine now.

In your solution you are just adding Any.

Thanks for your feedback! Just wanted to clarify that the example I used was only for illustration purposes but it might have given a wrong idea about the approach. I truly appreciate your advice, and I've added a subgoal to "Introduce the correct type arguments in different parts of the code" to make the approach more clear.

Also, I have enabled commenting so that you can give me feedback more easily, and I would be happy to work on it. Thanks again for your guidance!😊

[RyanWalker277]

Hey! So, now that the proposal submission window for GSOC'23 is closed, I just wanted to say thanks to @sobolevn and @intgr for being awesome supporters throughout my time contributing to Django-Stubs. Seriously, you guys rock! Whether or not I get selected for the program, I'm just grateful for everything I've learned from you both. Your guidance and encouragement have been invaluable to me. I'm definitely going to keep contributing to this project and try to make it even better. So, thanks again for being amazing mentors and helping me grow in so many ways!

[sobolevn]

I really hope to work together on the next things :)

[RyanWalker277]

So, I didn't make the cut for GSOC this year. It stings a little, but I'm still grateful for all the support and help that @sobolevn and @intgr gave me throughout the whole application process. Those guys are rockstars! I'm not gonna let this setback stop me from contributing to this project. I'm gonna push myself even harder and will try to get involved in contributing to core Django. Thanks again for having my back, guys. Let's keep crushing it together! Cheers! 🍻

[auvipy]

slightly off topic: I am considering celery stub instead of annotating the existing codes

[adamchainz]

Thanks for submitting @RyanWalker277 . Ultimately we had too many submissions and too few mentors, so we had to pick. Any ongoing contributions are welcome.