Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lock files don't mesh with dynamic package version info #7533

Closed
hynek opened this issue Sep 19, 2024 · 62 comments · Fixed by #10622
Closed

Lock files don't mesh with dynamic package version info #7533

hynek opened this issue Sep 19, 2024 · 62 comments · Fixed by #10622
Labels
needs-decision Undecided if this should be done needs-design Needs discussion, investigation, or design

Comments

@hynek
Copy link
Contributor

hynek commented Sep 19, 2024

Since tox-uv just shipped lock files, I took a stab at moving attrs to a fully-locked dev environment.

Here's the experiment PR: python-attrs/attrs#1349

A problem I've run into is that packages often use dynamic packaging data that is based on git metadata (setuptools-scm, hatch-vcs, etc). As such, attrs's package version changes after every commit, which allows us to upload to test PyPI and continually verify our package. See, for example, https://test.pypi.org/project/attrs/24.2.1.dev19/

Unfortunately the current project gets locked like this:

[[package]]
name = "attrs"
version = "24.2.1.dev20"
source = { editable = "." }

Which means it's outdated after each commit, because every commit increments the number behind dev.

Is there a way around this that I've missed? If not, could there be built one? 😇 AFAICT, this is currently the biggest blocker for FOSS use from uv's side. I'm also not sure what the point of that version lock is in the first place for the current, editable project? But that's a topic for another day. :)


❯ uv --version
uv 0.4.12 (2545bca69 2024-09-18)
@sbidoul
Copy link

sbidoul commented Sep 19, 2024

@hynek have you seen tool.uv.cache-keys ?

@hynek
Copy link
Contributor Author

hynek commented Sep 19, 2024

I have while searching this bug tracker, but I couldn't find a way for it to affect uv.lock? Changing the value to cache-keys = [{ git = true }, { file = "pyproject.toml" }] at least doesn't do anything when running uv.lock.

From the documentation (and its name 🤓) I've gathered it's mostly about how caching, not about locking? And TBH it sounds to me as if adding git as a git key, I'm actually adding cache busting.

@charliermarsh
Copy link
Member

I think you could pass --upgrade-package attrs to ensure that attrs is rebuilt and that the lockfile is updated. Or even set it in your configuration:

[tool.uv]
upgrade-package = ["attrs"]

But there's sort of an infinite loop here, right? Since every time you commit the lockfile, it will be outdated. So you then re-lock, and commit again, etc. I don't know how to solve this holistically. We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

@hynek
Copy link
Contributor Author

hynek commented Sep 20, 2024

But there's sort of an infinite loop here, right? Since every time you commit the lockfile, it will be outdated. So you then re-lock, and commit again, etc. We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

Yeah that's what I meant in my last sentence, but I'm always assuming Chesterton's Fence. :) Setting it to dynamic seems the best way since it maps nicely to project.dynamic. Just dunno if it would be a problem that there's a case where there's an invalid version "number" in the field which would speak for omitting.

@jkeifer
Copy link

jkeifer commented Sep 20, 2024

I too have this problem. I generally subscribe to the philosophy that I use a VCS for versioning, thus what is in the VSC repo shouldn't track it's own version because that's what I'm using the VCS for. Said another way, at best a version tracked within repo is imperfect for exactly the reasons described above about how you can't update a version from a commit without making another commit, i.e., a new version.

I'd love to understand the rationale behind storing this version in the lock file. Can anyone elaborate on that?

@burgholzer
Copy link

We are also facing a similar problem over at cda-tum/mqt-core#706 where we use renovate to keep the uv lock file up to date, which would be pretty convenient in principle.
However, currently it triggers a never ending chain of renovate update PRs because every update to the lock file adds a commit, which changes the setuptools-scm version, which causes another update PR to be triggered. (see cda-tum/mqt-core#703, cda-tum/mqt-core#704, cda-tum/mqt-core#705, cda-tum/mqt-core#706)

@LordFckHelmchen
Copy link

This problem is not only for dynamic versioning. We have a release-please workflow, which will create a PR for the next release. This will update the pyproject.toml version correctly but of course does not know about the uv.lock version.
If there would be an option to omit the project version from the uv.lock it would be great, as I currently also don't understand why it needs to be kept in the lock-file itself.

@david-waterworth
Copy link

As I mentioned in ttps://github.com//issues/7994 I usepython-semantic-release which behaves the same. The way I work around the infinite build loop is to configure my Jenkins script to skip commits that contains the python-semantic-release message - these commits include the changed pyproject.toml and __version__.py file, along with a generated CHANGELOG.md

As a workaround, I also added uv lock at the start of the build script and git add uv.lock at the end - this ensures that python-semantic-release commit includes the correctly versioned uv.lock file (I'm not sure yet if this causes other problems, i.e. updating other packages - ideally there would be a way of only updating the package version in uv.lock and nothing else.

@alejandrodnm
Copy link

I've just encountered this. We are also using release-please and uv in an internal project, after a release then the surprise is that the uv.lock was outdated.

Ours plans were to add uv to https://github.com/timescale/pgai but since we are already using release-please, we are hesitant due to this issue.

@RazerM
Copy link

RazerM commented Nov 6, 2024

While the setuptools-scm case seems difficult to solve, I think another common reason1 a dynamic version is used is when the version is statically defined but not in pyproject.toml. Typically as a __version__ attribute like in this example:

  1. Create example project

    uv init --lib mpypkg && cd mpypkg
  2. Configure pyproject.toml to use a dynamic version

    cat <<EOF > pyproject.toml
    [build-system]
    requires = ["setuptools"]
    build-backend = "setuptools.build_meta"
    
    [project]
    name = "mpypkg"
    requires-python = ">=3.12"
    dynamic = ["version"]
    
    [tool.uv]
    cache-keys = [{ file = "pyproject.toml" }, { file = "src/mpypkg/__init__.py" }]
    
    [tool.setuptools.dynamic]
    version = { attr = "mpypkg.__version__" }
    
    [tool.setuptools]
    package-dir = { "" = "src" }
    
    [tool.setuptools.packages.find]
    where = ["src"]
    EOF
  3. Add __version__ attribute to __init__.py

    cat <<EOF >> src/mpypkg/__init__.py
    __version__ = "1.0"
    EOF
  4. Run uv lock

  5. Change version number

    sed -i.bak 's/__version__ = "1\.0"/__version__ = "1.1"/' src/mpypkg/__init__.py
  6. Run uv lock

What I expected:

 version = 1
 requires-python = ">=3.12"
 
 [[package]]
 name = "mpypkg"
-version = "1.0"
+version = "1.1"
 source = { editable = "." }

What happened: uv.lock still has version = "1.0"

I've also tried uv lock --no-cache --refresh to no avail.

The version doesn't change until I do an unrelated uv add or something. This seems like a cache invalidation problem more than anything else, which I expected cache-keys to solve.


❯ uv --version
uv 0.4.30 (61ed2a236 2024-11-04)

Footnotes

  1. sometimes for backward compatibility reasons or in my case I can't move version into pyproject.toml until some other tooling I use catches up.

@charliermarsh
Copy link
Member

We can fix this, but I still strongly recommend against setting the version dynamically like that.

@charliermarsh
Copy link
Member

I've fixed the issue @RazerM pointed out in #8867. I guess I'll leave this issue open though since it's more about the inherent incompatibility between lockfiles and VCS-based versioning.

@cpascual
Copy link
Contributor

cpascual commented Nov 6, 2024

Hi @charliermarsh, you mention "inherent incompatibility of lock files and VCS-based versioning" , but you also mentioned:

We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

To me the above workaround seems good enough, specially if it is controlled by an opt-in configuration (e.g. omit_lock_version=["mypackage",]).

Is there any fundamental problem with this approach?

@cpascual
Copy link
Contributor

cpascual commented Nov 6, 2024

my use case is that I use setuptools-scm and the presence of commit-dependent version numbers in uv.lock is a frustrating cause of git merge conflicts preventing what otherwise would be an automated merge.

@ceejatec
Copy link

ceejatec commented Nov 8, 2024

We can fix this, but I still strongly recommend against setting the version dynamically like that.

@charliermarsh Out of curiosity, why?

@Coruscant11
Copy link
Contributor

Coruscant11 commented Nov 8, 2024

I just discovered this issue with the recent release 0.5.0. Using --locked when syncing the project makes everything fails. I have a dynamic version based on git tags, and when we release we tag our repository. So the lock file can be outdated without any code change.

I think excluding editable dependencies in the lock file could be a good workaround. Or we can also write "dynamic" as a version. By the way I will check what tools like poetry for example are doing in this use case.
@charliermarsh Is there any news on this? In the meantime, we will pin our version to the 0.4.30.

By the way I would suggest to list it in the breaking changes of the release

Thanks !!!

Edit: Forgot to mention, I was wondering if you already have some implementation ideas, I totally volunteer to work on this if needed. I will try a draft pull-request on my side for fun

@Stealthii
Copy link

We can fix this, but I still strongly recommend against setting the version dynamically like that.

If references or opinion articles exist around this topic, they are few in number. I think a few on this issue would be interested in wider discussion around this. PEP 621 introduced toml metadata, and as of writing the official specification still allows version to be set dynamically, with no recommendation to the contrary.

It's a pretty standard paradigm (see hatch-vcs, setuptools-scm, poetry-dynamic-versioning) for single project repos where versioning, point release branches, development builds, etc. are all managed and controlled via VCS change control.

Whilst alternatives exist for separating static versions (such as 0.5.1 from a generated identifier ref like f399a5271 in uv 0.5.1 (f399a5271 2024-11-08)), there are some compelling reasons to use dynamic (such as dunamai) formed versions:

  • Unique reference of all release, post, dev, dirty distributions of Python packages
  • All tarball/wheel builds being uniquely available in an internal PyPI repository for further testing or triage
  • Rules around versioning and releases can be set by project owners irrespective of development language
  • No room for user error (such as forgetting to update a static version of 0.5.1 for the git tag v0.5.1 and publishing builds)
  • Exposing the version in test logs, headers, etc. automatically shows the exact build used, aiding debugging

My expectation for the project package that has a dynamic version (source = { editable = "." } specifically) would be simply not defining version in uv.lock for the local package (or providing a dynamic reference). This by nature isn't a lockable reference (especially when set by VCS signatures) whereas other uses of uv lock such as multi-project mono repos or static versions uphold the current behaviour.

It's worth noting that although version is defined dynamically in source control, builds of the package provide the version statically:

ls -1 dist/*
dist/mypackage-0.5.0.dev5+g5d056c1-py3-none-any.whl
dist/mypackage-0.5.0.dev5+g5d056c1.tar.gzunzip -p dist/*.whl '*.dist-info/METADATA' | grep -E '^Version: '
Version: 0.5.0.dev5+g5d056c1tar -Oxf dist/*.tar.gz '*.egg-info/PKG-INFO' | grep -E '^Version: '
Version: 0.5.0.dev5+g5d056c1

@charliermarsh
Copy link
Member

If references or opinion articles exist around this topic, they are few in number.

This is just my personal opinion! Dynamic metadata makes a number of things more complicated than they need to be. I understand the use-case for scm versioning, but in my opinion using dynamic metadata to read a version from __init__.py is a really heavy hammer with the only benefit being that you get to avoid writing out a constant twice -- I was responding to that specific case, where I don't think the tradeoffs are very good.

@charliermarsh
Copy link
Member

...simply not defining version in uv.lock for the local package (or providing a dynamic reference)

Unfortunately it's not super simple -- it breaks the fundamental assumption that packages have versions, so we have to take that into account everywhere.

@Coruscant11
Copy link
Contributor

Unfortunately it's not super simple -- it breaks the fundamental assumption that packages have versions, so we have to take that into account everywhere.

Yes this is what I saw this week-end when I was trying to contribute on this 🥲
But anyway, I had the opportunity to deep dive in the code base so it's fine.
We will have to trust you on this 😄 (No need to worry at all then)

Thanks!

@ceejatec
Copy link

it breaks the fundamental assumption that packages have versions

Not exactly, I don't think. The package always DOES have a version. It's just that the version may change outside of uv's immediate notice. Perhaps it would be fine if uv lock and uv sync and similar commands re-assessed the version if it's marked dynamic in the pyproject.toml. That wouldn't handle cases where the dynamic version contains the git commit SHA in some form, but I suspect that's pretty much an intractable problem.

That said, I feel like a cleaner solution would be to omit the main package from uv.lock entirely. However I'm definitely not familiar with the implementation here, so I don't know what unforeseen consequences that might have.

@hynek
Copy link
Contributor Author

hynek commented Nov 13, 2024

If references or opinion articles exist around this topic, they are few in number.

This is just my personal opinion! Dynamic metadata makes a number of things more complicated than they need to be. I understand the use-case for scm versioning, but in my opinion using dynamic metadata to read a version from __init__.py is a really heavy hammer with the only benefit being that you get to avoid writing out a constant twice -- I was responding to that specific case, where I don't think the tradeoffs are very good.

By the way, this can be inverted and made more standard-compliant while doing so by putting something like this into your __init__.py:

def __getattr__(name: str) -> str:
    if name != "__version__":
        msg = f"module {__name__} has no attribute {name}"
        raise AttributeError(msg)

    from importlib.metadata import version

    return version("YOUR-PACKAGE")

You get static metadata with no duplication.

@ninoseki
Copy link

I have the same pain point and would like to share how many projects in the wild do Git based versioning for your information.

@roteiro
Copy link

roteiro commented Nov 13, 2024

I have the same pain point and would like to share how many projects in the wild do Git based versioning for your information.

add to that:

@ofek
Copy link
Contributor

ofek commented Jan 8, 2025

The solution here is what should be preferred for hatch-vcs, setuptools-scm, etc. Basically, only temporarily set the SETUPTOOLS_SCM_PRETEND_VERSION environment variable when running uv lock.

@charliermarsh
Copy link
Member

I'm working on omitting dynamic versions from the lockfile: #10622

@namurphy
Copy link
Contributor

namurphy commented Jan 15, 2025

Many thanks for making this change and including it in v0.15.9! 🎉🎂💖🎆🪩🌌🥦

I just recreated uv.lock for a project with a dynamic version, and can confirm that the resulting uv.lock no longer contains the dynamic version for the project. I deleted the prior uv.lock before running uv lock with v0.15.9, just to be safe.

@brendan-morin
Copy link

Perhaps I am misunderstanding the fix action, using this improperly, or this is a hatch-vcs problem, but are we sure this is working as intended? It seems that the behavior is inconsistent and flip flops depending on the state of uv's locking/my venv. Please let me know if this is better suited for a separate issue thread.

uv --version
uv 0.5.24 (Homebrew 2025-01-23)

I have a pyproject.toml like this:

[project]
dynamic = ["version"]

[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

[tool.hatch.version]
source = "vcs"

As a caveat, I experienced a huge amount of inconsistency in reproduction (generally almost always ending in a state where the version was included in the uv.lock file), so I am not 100% confident these steps will repro for others. But this is what I used to repro my observed behavior.

  1. delete uv.lock + .venv
  2. run uv lock. This initially works as expected, no dynamic version in uv.lock
  3. run uv sync. Still fine, no dynamic version in uv.lock. But we see the package has been built according to the logs:
$ uv sync
Resolved 108 packages in 1ms
   Built mypackage @ file:///...
Prepared 2 packages in 1.55s
Uninstalled 2 packages in 57ms
Installed 2 packages in 9ms
 ...
 ~ mypackage==5.1.29.dev1+gdd9890e.d20250128 (from file:///...)
  1. run uv lock again. Now we see these new logs:
$ uv lock
Resolved 108 packages in 30ms
Updated mypackage (dynamic) -> v5.1.29.dev1+gdd9890e.d20250128

And the dynamic version is present in the lock file:

[[package]]
name = "mypackage"
version = "5.1.29.dev1+gdd9890e.d20250128"
  1. Now no matter how many times I delete uv.lock, venv, run uv lock, or uv sync, the dynamic version remains. There's probably other ways, but if I want to reset the process, I can add this to pyproject.toml and rerun uv lock:
[tool.hatch.build.hooks.vcs]
version-file = "_version.py" 

and uv lock yields:

uv lock
Resolved 108 packages in 1m 42s
Updated mypackage v5.1.29.dev1+gdd9890e.d20250128 -> (dynamic)
  1. Repeat the process by running uv sync and uv lock again:
$ uv sync            
Resolved 108 packages in 1ms
   Built mypackage @ file:///...
Prepared 1 package in 945ms
Uninstalled 1 package in 0.58ms
Installed 1 package in 1ms
 ~ mypacakge==5.1.29.dev1+gdd9890e.d20250128 (from file:///...)
$ uv lock
Resolved 108 packages in 3.25s
Updated mypackage (dynamic) -> v5.1.29.dev1+gdd9890e.d20250128

@charliermarsh
Copy link
Member

That seems unusual. Are you able to put together a project that I can use to reproduce?

@apollo13
Copy link

I am seeing the same but having a hard time reproducing why this is happening. In my case it is enough to remove uv.lock and run uv lock again. Any debug logs that might have anything helpful?

@apollo13
Copy link

Okay, uv lock -vvv showed stuf like:

 uv_distribution::distribution_database::get_or_build_wheel_metadata dist=oversight @ file:///home/florian/dev/projectx/oversight
    0.014659s   0ms DEBUG uv_distribution::source No static `pyproject.toml` available for: oversight @ file:///home/florian/dev/projectx/oversight (PyprojectToml(DynamicField("version")))
    0.015007s DEBUG uv_fs Acquired lock for `/home/florian/.cache/uv/sdists-v6/editable/5bff42cdde91dde7`
    0.015906s   1ms DEBUG uv_distribution::source Using cached metadata for: oversight @ file:///home/florian/dev/projectx/oversight
    0.016398s   2ms DEBUG uv_workspace::workspace No workspace root found, using project root
    0.016577s   2ms DEBUG uv_fs Released lock at `/home/florian/.cache/uv/sdists-v6/editable/5bff42cdde91dde7/.lock`
 uv_resolver::resolver::solve 
    0.018200s   0ms DEBUG uv_resolver::resolver Solving with installed Python version: 3.13.1
    0.018216s   0ms DEBUG uv_resolver::resolver Solving with target Python version: >=3.13.0, <3.14
   uv_resolver::resolver::get_dependencies_forking package=root, version=0a0.dev0
     uv_resolver::resolver::get_dependencies package=root, version=0a0.dev0
    0.018550s   0ms DEBUG uv_resolver::resolver Adding direct dependency: oversight[container]*
    0.018629s   0ms DEBUG uv_resolver::resolver Searching for a compatible version of oversight @ file:///home/florian/dev/projectx/oversight (*)
   uv_resolver::resolver::get_dependencies_forking package=oversight[container], version=0.4.1.dev31+ga536f6c.d20250128
     uv_resolver::resolver::get_dependencies package=oversight[container], version=0.4.1.dev31+ga536f6c.d20250128
    0.018700s   0ms DEBUG uv_resolver::resolver Adding direct dependency: oversight==0.4.1.dev31+ga536f6c.d20250128
    0.018724s   0ms DEBUG uv_resolver::resolver Adding direct dependency: oversight[container]==0.4.1.dev31+ga536f6c.d20250128
    0.018761s   0ms DEBUG uv_resolver::resolver Searching for a compatible version of oversight @ file:///home/florian/dev/projectx/oversight (==0.4.1.dev31+ga536f6c.d20250128)
   uv_resolver::resolver::get_dependencies_forking package=oversight, version=0.4.1.dev31+ga536f6c.d20250128
     uv_resolver::resolver::get_dependencies package=oversight, version=0.4.1.dev31+ga536f6c.d20250128
    0.018960s   0ms DEBUG uv_resolver::resolver Adding direct dependency: dj-database-url>=2.2.0

After running a uv cache clean uv lock no longer seems to write the version into uv.lock. Now running uv sync and removing uv.lock and rerunning uv lock again writes it back.

So I wonder if this depends on the cache?

@nathanjmcdougall
Copy link
Contributor

nathanjmcdougall commented Jan 28, 2025

I'm having a similar issue. To provide another data point, I have a project where:

  1. Using the project created pre-v0.15.9 will fail to use remove the version from the lockfile, even when deleting the lockfile and .venv.
  2. A fresh clone and running uv lock will remove the version from the lock file no problem.

So a cache issue seems likely to me.

@neutrinoceros
Copy link

neutrinoceros commented Jan 28, 2025

@brendan-morin Maybe a bit random but I noticed that your [project] table is lacking a requires-python entry, which seemed to be crucial in my encounter with a similar problem.

xref astral-sh/uv-pre-commit#35

@charliermarsh
Copy link
Member

Is anyone able to upload a project that I can use to reproduce this? I'd love to fix it. I'm happy to fix it tonight.

@apollo13
Copy link

@charliermarsh Use this repo: https://github.com/apollo13/uv-bug

Follow this to the letter (I am not sure if you can repro if you do anything different):

  • uv cache clean
  • uv lock and observe that git status shows no changes
  • uv sync
  • rm -rf uv.lock .venv
  • uv lock
  • Run git diff and see the version written into the lock file (shouldn't be)
  • uv cache clean
  • rm -rf uv.lock .venv
  • uv lock (No version written -> okay)

@charliermarsh
Copy link
Member

Thanks!

@charliermarsh
Copy link
Member

Thank you, this was a real bug. Fix here: #11046.

@brendan-morin
Copy link

Awesome, thank you for the quick repro @apollo13 and @charliermarsh for the quick fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-decision Undecided if this should be done needs-design Needs discussion, investigation, or design
Projects
None yet
Development

Successfully merging a pull request may close this issue.