Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce lockfile schema versions #8509

Merged
merged 1 commit into from
Oct 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion crates/uv-resolver/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ pub use exclude_newer::ExcludeNewer;
pub use exclusions::Exclusions;
pub use flat_index::{FlatDistributions, FlatIndex};
pub use lock::{
Lock, LockError, RequirementsTxtExport, ResolverManifest, SatisfiesResult, TreeDisplay,
Lock, LockError, LockVersion, RequirementsTxtExport, ResolverManifest, SatisfiesResult,
TreeDisplay, VERSION,
};
pub use manifest::Manifest;
pub use options::{Flexibility, Options, OptionsBuilder};
Expand Down
22 changes: 21 additions & 1 deletion crates/uv-resolver/src/lock/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ mod requirements_txt;
mod tree;

/// The current version of the lockfile format.
const VERSION: u32 = 1;
pub const VERSION: u32 = 1;

static LINUX_MARKERS: LazyLock<MarkerTree> = LazyLock::new(|| {
MarkerTree::from_str(
Expand Down Expand Up @@ -494,6 +494,11 @@ impl Lock {
self
}

/// Returns the lockfile version.
pub fn version(&self) -> u32 {
self.version
}

/// Returns the number of packages in the lockfile.
pub fn len(&self) -> usize {
self.packages.len()
Expand Down Expand Up @@ -1509,6 +1514,21 @@ impl TryFrom<LockWire> for Lock {
}
}

/// Like [`Lock`], but limited to the version field. Used for error reporting: by limiting parsing
/// to the version field, we can verify compatibility for lockfiles that may otherwise be
/// unparsable.
#[derive(Clone, Debug, serde::Deserialize)]
pub struct LockVersion {
version: u32,
}

impl LockVersion {
/// Returns the lockfile version.
pub fn version(&self) -> u32 {
self.version
}
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Package {
pub(crate) id: PackageId,
Expand Down
10 changes: 6 additions & 4 deletions crates/uv/src/commands/project/add.rs
Original file line number Diff line number Diff line change
Expand Up @@ -556,8 +556,8 @@ pub(crate) async fn add(

// Update the `pypackage.toml` in-memory.
let project = project
.with_pyproject_toml(toml::from_str(&content).map_err(ProjectError::TomlParse)?)
.ok_or(ProjectError::TomlUpdate)?;
.with_pyproject_toml(toml::from_str(&content).map_err(ProjectError::PyprojectTomlParse)?)
.ok_or(ProjectError::PyprojectTomlUpdate)?;

// Set the Ctrl-C handler to revert changes on exit.
let _ = ctrlc::set_handler({
Expand Down Expand Up @@ -758,8 +758,10 @@ async fn lock_and_sync(

// Update the `pypackage.toml` in-memory.
project = project
.with_pyproject_toml(toml::from_str(&content).map_err(ProjectError::TomlParse)?)
.ok_or(ProjectError::TomlUpdate)?;
.with_pyproject_toml(
toml::from_str(&content).map_err(ProjectError::PyprojectTomlParse)?,
)
.ok_or(ProjectError::PyprojectTomlUpdate)?;

// Invalidate the project metadata.
if let VirtualProject::Project(ref project) = project {
Expand Down
48 changes: 38 additions & 10 deletions crates/uv/src/commands/project/lock.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ use std::collections::BTreeSet;
use std::fmt::Write;
use std::path::Path;

use anstream::eprint;
use owo_colors::OwoColorize;
use rustc_hash::{FxBuildHasher, FxHashMap};
use tracing::debug;
Expand All @@ -28,8 +27,8 @@ use uv_python::{Interpreter, PythonDownloads, PythonEnvironment, PythonPreferenc
use uv_requirements::upgrade::{read_lock_requirements, LockedRequirements};
use uv_requirements::ExtrasResolver;
use uv_resolver::{
FlatIndex, InMemoryIndex, Lock, Options, OptionsBuilder, PythonRequirement, RequiresPython,
ResolverManifest, ResolverMarkers, SatisfiesResult,
FlatIndex, InMemoryIndex, Lock, LockVersion, Options, OptionsBuilder, PythonRequirement,
RequiresPython, ResolverManifest, ResolverMarkers, SatisfiesResult, VERSION,
};
use uv_types::{BuildContext, BuildIsolation, EmptyInstalledPackages, HashStrategy};
use uv_warnings::{warn_user, warn_user_once};
Expand Down Expand Up @@ -204,7 +203,15 @@ pub(super) async fn do_safe_lock(
Ok(result)
} else {
// Read the existing lockfile.
let existing = read(workspace).await?;
let existing = match read(workspace).await {
Ok(Some(existing)) => Some(existing),
Ok(None) => None,
Err(ProjectError::Lock(err)) => {
warn_user!("Failed to read existing lockfile; ignoring locked requirements: {err}");
None
}
Err(err) => return Err(err),
};

// Perform the lock operation.
let result = do_lock(
Expand Down Expand Up @@ -903,13 +910,34 @@ async fn commit(lock: &Lock, workspace: &Workspace) -> Result<(), ProjectError>
/// Returns `Ok(None)` if the lockfile does not exist.
pub(crate) async fn read(workspace: &Workspace) -> Result<Option<Lock>, ProjectError> {
match fs_err::tokio::read_to_string(&workspace.install_path().join("uv.lock")).await {
Ok(encoded) => match toml::from_str(&encoded) {
Ok(lock) => Ok(Some(lock)),
Err(err) => {
eprint!("Failed to parse lockfile; ignoring locked requirements: {err}");
Ok(None)
Ok(encoded) => {
match toml::from_str::<Lock>(&encoded) {
Ok(lock) => {
// If the lockfile uses an unsupported version, raise an error.
if lock.version() != VERSION {
return Err(ProjectError::UnsupportedLockVersion(
VERSION,
lock.version(),
));
}
Ok(Some(lock))
}
Err(err) => {
// If we failed to parse the lockfile, determine whether it's a supported
// version.
if let Ok(lock) = toml::from_str::<LockVersion>(&encoded) {
if lock.version() != VERSION {
return Err(ProjectError::UnparsableLockVersion(
VERSION,
lock.version(),
err,
));
}
}
Err(ProjectError::UvLockParse(err))
}
}
},
}
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(err) => Err(err.into()),
}
Expand Down
13 changes: 11 additions & 2 deletions crates/uv/src/commands/project/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,12 @@ pub(crate) enum ProjectError {
)]
MissingLockfile,

#[error("The lockfile at `uv.lock` uses an unsupported schema version (v{1}, but only v{0} is supported). Downgrade to a compatible uv version, or remove the `uv.lock` prior to running `uv lock` or `uv sync`.")]
UnsupportedLockVersion(u32, u32),

#[error("Failed to parse `uv.lock`, which uses an unsupported schema version (v{1}, but only v{0} is supported). Downgrade to a compatible uv version, or remove the `uv.lock` prior to running `uv lock` or `uv sync`.")]
UnparsableLockVersion(u32, u32, #[source] toml::de::Error),

#[error("The current Python version ({0}) is not compatible with the locked Python requirement: `{1}`")]
LockedPythonIncompatibility(Version, RequiresPython),

Expand Down Expand Up @@ -128,11 +134,14 @@ pub(crate) enum ProjectError {
#[error("Project virtual environment directory `{0}` cannot be used because {1}")]
InvalidProjectEnvironmentDir(PathBuf, String),

#[error("Failed to parse `uv.lock`")]
UvLockParse(#[source] toml::de::Error),

#[error("Failed to parse `pyproject.toml`")]
TomlParse(#[source] toml::de::Error),
PyprojectTomlParse(#[source] toml::de::Error),

#[error("Failed to update `pyproject.toml`")]
TomlUpdate,
PyprojectTomlUpdate,

#[error(transparent)]
Python(#[from] uv_python::Error),
Expand Down
94 changes: 94 additions & 0 deletions crates/uv/tests/it/lock.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14913,6 +14913,100 @@ fn lock_invalid_project_table() -> Result<()> {
Ok(())
}

#[test]
fn lock_unsupported_version() -> Result<()> {
let context = TestContext::new("3.12");

let pyproject_toml = context.temp_dir.child("pyproject.toml");
pyproject_toml.write_str(
r#"
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["iniconfig==2.0.0"]

[build-system]
requires = ["setuptools>=42"]
build-backend = "setuptools.build_meta"
"#,
)?;

// Validate schema, invalid version.
context.temp_dir.child("uv.lock").write_str(
r#"
version = 2
requires-python = ">=3.12"

[options]
exclude-newer = "2024-03-25T00:00:00Z"

[[package]]
name = "iniconfig"
version = "2.0.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/d7/4b/cbd8e699e64a6f16ca3a8220661b5f83792b3017d0f79807cb8708d33913/iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3", size = 4646 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374", size = 5892 },
]

[[package]]
name = "project"
version = "0.1.0"
source = { editable = "." }
dependencies = [
{ name = "iniconfig" },
]

[package.metadata]
requires-dist = [{ name = "iniconfig", specifier = "==2.0.0" }]
"#,
)?;

uv_snapshot!(context.filters(), context.lock().arg("--frozen"), @r###"
success: false
exit_code: 2
----- stdout -----

----- stderr -----
error: The lockfile at `uv.lock` uses an unsupported schema version (v2, but only v1 is supported). Downgrade to a compatible uv version, or remove the `uv.lock` prior to running `uv lock` or `uv sync`.
"###);

// Invalid schema (`iniconfig` is referenced, but missing), invalid version.
context.temp_dir.child("uv.lock").write_str(
r#"
version = 2
requires-python = ">=3.12"

[options]
exclude-newer = "2024-03-25T00:00:00Z"

[[package]]
name = "project"
version = "0.1.0"
source = { editable = "." }
dependencies = [
{ name = "iniconfig" },
]

[package.metadata]
requires-dist = [{ name = "iniconfig", specifier = "==2.0.0" }]
"#,
)?;

uv_snapshot!(context.filters(), context.lock().arg("--frozen"), @r###"
success: false
exit_code: 2
----- stdout -----

----- stderr -----
error: Failed to parse `uv.lock`, which uses an unsupported schema version (v2, but only v1 is supported). Downgrade to a compatible uv version, or remove the `uv.lock` prior to running `uv lock` or `uv sync`.
Caused by: Dependency `iniconfig` has missing `version` field but has more than one matching package
"###);

Ok(())
}

/// See: <https://github.com/astral-sh/uv/issues/7618>
#[test]
fn lock_change_requires_python() -> Result<()> {
Expand Down
18 changes: 18 additions & 0 deletions docs/concepts/resolution.md
Original file line number Diff line number Diff line change
Expand Up @@ -397,3 +397,21 @@ reading and extracting archives in the following formats:

For more details about the internals of the resolver, see the
[resolver reference](../reference/resolver-internals.md) documentation.

## Lockfile versioning

The `uv.lock` file uses a versioned schema. The schema version is included in the `version` field of
the lockfile.

Any given version of uv can read and write lockfiles with the same schema version, but will reject
lockfiles with a greater schema version. For example, if your uv version supports schema v1,
`uv lock` will error if it encounters an existing lockfile with schema v2.

uv versions that support schema v2 _may_ be able to read lockfiles with schema v1 if the schema
update was backwards-compatible. However, this is not guaranteed, and uv may exit with an error if
it encounters a lockfile with an outdated schema version.

The schema version is considered part of the public API, and so is only bumped in minor releases, as
a breaking change (see [Versioning](../reference/versioning.md)). As such, all uv patch versions
within a given minor uv release are guaranteed to have full lockfile compatibility. In other words,
lockfiles may only be rejected across minor releases.
11 changes: 11 additions & 0 deletions docs/reference/versioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,14 @@ uv does not yet have a stable API; once uv's API is stable (v1.0.0), the version
adhere to [Semantic Versioning](https://semver.org/).

uv's changelog can be [viewed on GitHub](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md).

## Cache versioning

Cache versions are considered internal to uv, and so may be changed in a minor or patch release. See
[Cache versioning](../concepts/cache.md#cache-versioning) for more.

## Lockfile versioning

The `uv.lock` schema version is considered part of the public API, and so will only be incremented
in a minor release as a breaking change. See
[Lockfile versioning](../concepts/resolution.md#lockfile-versioning) for more.
Loading