Migrate to uv as package manager for the generated project

[foarsitter]

The project itself now uses uv and this is an attempt to bring uv to the generated project too.

• Python commands are executed through uv run for local development. In Docker the .venv python interpreter is added to the $PATH so packages can be used system wide.
• The uv.lock file is empty on start because is is updated when uv sync is executed.
• For Heroku I added a pre-commit that compiles a requirements.txt
• Added a trap to test_docker.sh to cleanup after testing so the next time will no fail due to an existing postgres volume.
• In test_docker.sh I added a step to test building the docker production image

Before we can merge this we need one last round to update all the versions to the latest ones in the requirement.txt files.

[foarsitter]

Related to #3083 #1988

[foarsitter]

Ready for testing 🎉

cookiecutter https://github.com/foarsitter/cookiecutter-django --checkout uv-generated-project

[browniebroke]

Did a first pass and noted a few things. Haven't reviewed the Dockerfiles yet

[steveputman]

Hi, there. I played around with using your fork to update some existing projects and eventually got them working but ended up reverting them to their production (pip) versions so that I could revisit when I had more time.

When working on an adapting an existing project, I got an error during the build, which was reproduced when I built a project from scratch using cookiecutter https://github.com/foarsitter/cookiecutter-django --checkout uv-generated-project:

[django python-build-stage 4/6] RUN --mount=type=cache,target=/root/.cache/uv --mount=type=bind,source=uv.lock,target=uv.lock --mount=type=bind,source=pyproject.toml,target=pyproject.toml uv sync --frozen --no-install-project --no-dev:
0.112 Using CPython 3.12.7 interpreter at: /usr/local/bin/python
0.112 Creating virtual environment at: .venv
0.113 error: Could not find root package my-awesome-project
'

I believe that it's caused by having the blank uv.lock. I thought since the local project had the host file rw that it would generate a usable uv.lock during the local build. Is there a way to get a usable version of uv.lock on the host filesystem without running uv sync on the host filesystem? Thanks for this--very cool!

[foarsitter]

@steveputman thanks looking into this!

During a build you cannot write to the host system.

Your interpreter is inside the container together with uv so you can run uv:

docker compose -f docker-compose.local.yml run django uv lock

Struggled a lot with this and it is not ideal. So hopefully someone has a better solution.

[steveputman]

@foarsitter
The Docker documentation was sort of vague on that, but now I know.

Your solution is better than mine (I had uv installed on my host system so just ran uv lock in the project directory.

Thanks for the help! Will let you know if anything else comes up.

[rtpg]

I think you need to run uv sync on various start scripts (like celery), at least locally. Because you're mounting a folder to /app/.venv we actually don't have the installed virtualenv created in the base Dockerfile.

Of course we're also not sharing a virtualenv here, by my read/attempt to use this. It might be possible to just share a mount across all the services, but I'm having trouble getting that to work.

[pievalentin]

Great work! This project really need to have a better tool than the plain pip

[pievalentin]

The dependabot would need updating too. I dont know if it supports pyproject declarations

[leandrodesouzadev]

The dependabot would need updating too. I dont know if it supports pyproject declarations

I believe it does, I've found some discussions on the project, here's the link to the issue.

[browniebroke]

The dependabot would need updating too. I dont know if it supports pyproject declarations

This has been added now: dependabot/dependabot-core#10478 (comment)

[pievalentin]

a git pull away from a merge 😿

[foobacca]

Tried it. I wonder if the requirements currently in requirements/production.txt should be moved to an optional dependencies section in pyproject.toml, so something like:

[project.optional-dependencies]
production = [
  "gunicorn==23.0.0",
  # ...
]

Currently they appear in the main project dependencies, so they will be installed in the development virtualenv.

They can then be installed by running uv sync --extra=production - or explicitly excluded with uv sync --no-extra=production

uv docs about optional dependencies.

(And thanks for doing the work - I look forward to it being merged :) )

[foarsitter]

@foobacca I see development as a superset of production. Everything from production needs to be replicated in development but production doesn't need a test framework for example.

When the sentry_sdk is added as an extra dependency I cannot use capture_exception for example because the package is missing. Another example is testing the gunicorn command. I cannot do such thing when developing when I do not have the package.

[foobacca]

@foarsitter I see your point (though you can of course install the production dependencies if you want with uv sync --extra=production ).

I don't have a strong preference, but it does seem like a change from the master branch, where there is a separate requirements/production.txt file, suggesting that the maintainers liked to keep the production dependencies separate. So I wanted to make sure that change was explicit and intended.

I'll leave it to the maintainers. If they're happy with the change I have no objection. (And I'm quite capable of changing my project after running cookiecutter anyway 🙂 )

[foobacca]

One other idea. I would expect the [project] and [dependency-groups] section to be the first and second sections in pyproject.toml - putting the overall project definition and dependencies at the top where it's easy to find them.

I can obviously edit it myself after running cookiecutter, so not a big deal if others prefer it at the bottom, but thought I'd mention it.

[browniebroke]

I don't have a strong preference, but it does seem like a change from the master branch, where there is a separate requirements/production.txt file, suggesting that the maintainers liked to keep the production dependencies separate. So I wanted to make sure that change was explicit and intended.

The production deps are installed locally on the master branch too, so not so much of change - see #4838 for more context

[pievalentin]

One other idea. I would expect the [project] and [dependency-groups] section to be the first and second sections in pyproject.toml - putting the overall project definition and dependencies at the top where it's easy to find them.

I can obviously edit it myself after running cookiecutter, so not a big deal if others prefer it at the bottom, but thought I'd mention it.

I agree. Dependencies should be at the top of the file

[foarsitter]

We use https://github.com/tox-dev/pyproject-fmt for dictating the order of appearance, so you don't need to worry about it :)

[browniebroke]

Just realised that there is the question of PythonAnywhere too: https://cookiecutter-django.readthedocs.io/en/latest/3-deployment/deployment-on-pythonanywhere.html

Not sure what do about that:

cookiecutter-django/docs/3-deployment/deployment-on-pythonanywhere.rst

Lines 33 to 40 in 3524f6d

	.. code-block:: bash
	git clone <my-repo-url> # you can also use hg
	cd my-project-name
	mkvirtualenv --python=/usr/bin/python3.10 my-project-name
	pip install -r requirements/production.txt # may take a few minutes
	.. note:: We're creating the virtualenv using Python 3.10 (``--python=/usr/bin/python3.10```), although Cookiecutter Django generates a project for Python 3.12. This is because, at time of writing, PythonAnywhere only supports Python 3.10. It shouldn't be a problem, but if is, you may try changing the Python version to 3.12 and see if it works. If it does, please let us know, or even better, submit a pull request to update this section.

I haven't got much experience with this platform, do they support uv?

EDIT: Looks like they do

Meta: if you feel that pip is a bit too old-school for your package-management needs, you can try poetry, hatch, or uv.

[foarsitter]

Ok, I think we are ready to merge.

[browniebroke]

Nearly there 💪

[browniebroke]

Would be nice if we could address the latest comments but looking at them again, I don't think they're blockers.

[foarsitter]

Tests are failing now and I don't know why. Feel free to take over, need to fix something else first xD

[browniebroke]

Tests are failing now and I don't know why. Feel free to take over, need to fix something else first xD

I think it's something to do with the duplicated line ending in the pre-commit config... Not sure why but it passes when they're back

EDIT: I've just seen that you reverted that and it's still failing 😭

[baipaiha]

Failing tests seems related to docker container name conflict, like if test containers were not removed/clean.
Or not related at all ?

Error response from daemon: Conflict. The container name "/my_awesome_project_local_postgres" is already in use by container "3f4f06979b88e54037ac7ef161af59f2191fe50417e6877844024c94bcb25592". You have to remove (or rename) that container to be able to reuse that name.

[browniebroke]

I've changed this approach a bit by defining a minimal Docker image to run uv add commands and run it as a standalone Docker container (without compose involved). This is much faster to build and avoids having to start the other containers while generating the template, just for resolving dependencies.

[foarsitter]

I've changed this approach a bit by defining a minimal Docker image to run uv add commands and run it as a standalone Docker container (without compose involved). This is much faster to build and avoids having to start the other containers while generating the template, just for resolving dependencies.

Sounds like a good fundamental solution to me, thanks!

I tried it this way using the uv image directly but I couldn't get it to work. Looking to the Dockerfile I'm wondering why it works this way but I'm fine with it.

From my perspective we are ready to merge this as is!

[browniebroke]

From my perspective we are ready to merge this as is!

Let's do it! 🎉🎉🎉