From 2d9e9ef67743055d459ef4b4bcba880c2232cda2 Mon Sep 17 00:00:00 2001 From: pauleveritt Date: Tue, 30 Jul 2024 16:10:09 -0400 Subject: [PATCH 01/27] Include three comments from @carreau (drop two bullets, *args in example, explain *args.) --- peps/pep-0750.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index a6f4ce0e1d4..9be4553238f 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -97,7 +97,7 @@ Let's start with a tag string which simply returns a static greeting: .. code-block:: python - def greet(): + def greet(*args): """Give a static greeting.""" return "Hello!" @@ -147,9 +147,7 @@ strings: - ``args[0]`` is still the string-like ``'Hello '``, this time with a trailing space - ``args[1]`` is an expression -- the ``{name}`` part -- Tag strings represent this part as an *interpolation* object -- An interpolation is a tuple whose first item is a lambda -- Calling this lambda evaluates the expression in the original scope where the tag string was defined +- Tag strings represent this part as an *interpolation* object as discussed below The ``*args`` list is a sequence of ``Decoded`` and ``Interpolation`` values. A "decoded" object is a string-like object with extra powers, as described below. An "interpolation" object is a @@ -261,8 +259,9 @@ Evaluating Tag Strings ---------------------- When the tag string is evaluated, the tag must have a binding, or a ``NameError`` -is raised; and it must be a callable, or a ``TypeError`` is raised. This behavior -follows from the de-sugaring of: +is raised; and it must be a callable, or a ``TypeError`` is raised. The callable +must accept a sequence of positional arguments. This behavior follows from the +de-sugaring of: .. code-block:: python From f961b651ec1479e67439eb7dbb4b81df22962eb7 Mon Sep 17 00:00:00 2001 From: Jim Baker Date: Wed, 17 Jul 2024 16:01:10 -0600 Subject: [PATCH 02/27] Lambda-wrapped expressions use annotation scope --- peps/pep-0750.rst | 65 +++++++++++++++++++++++++++-------------------- 1 file changed, 38 insertions(+), 27 deletions(-) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index d36ac6b08c6..02bc743b87f 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -414,15 +414,15 @@ except that all expressions are always implicitly wrapped with a ``lambda``: lexical scope, including local and global variables. Any valid Python expression can be used, including function and method calls. -However, this PEP further proposes to extend the usual lexical scoping of a lambda-wrapped -expression to that supported by -`annotation scopes `_, -so as to minimize developer surprise. For the remainder of the section, *class scope* will be -used to refer to annotation scope as it's used by tag strings; this naming will be discussed -momentarily. +However, this PEP further proposes to extend the usual lexical scoping of a +lambda-wrapped expression in the interpolation, as seen with class definitions, +to that supported by `annotation scopes +`_, +so as to minimize developer surprise. -Let's look at why this is important with a somewhat contrived example where a developer sets up -how figures are rendered in a caption, using a ``html`` tag: +Let's look at why using annotation scope is important with a somewhat contrived +example where a developer sets up how figures are rendered in a caption, using a +``html`` tag: .. code-block:: python @@ -430,8 +430,11 @@ how figures are rendered in a caption, using a ``html`` tag: tag = 'b' figure = html'<{tag}>Figure' -If the scope is only the usual lexical scope for lambdas, when ``figure`` is finally evaluated, -the name ``tag`` is not available. Contrast that scoping with what is seen in using a f-string: +Unless the ``html`` function fully evaluates all interpolations before the class +is constructed, a subsequent interpolation will fail with ``NameError: name +'tag' is not defined``. This means that the name ``tag`` is no longer available +in the usual lexical scope for lambdas. Contrast that scoping with what is seen +in using a f-string: .. code-block:: python @@ -439,23 +442,31 @@ the name ``tag`` is not available. Contrast that scoping with what is seen in us tag = 'b' figure = f'<{tag}>Figure' -The class variable ``figure`` here does evaluate correctly, if at the risk of an HTML injection -attack. Because tag strings are supposed to work like f-strings, but with more capabilities, -it's necessary to support class scope. - -Why is it called class scope, instead of annotation scope? - -*Annotation* scope (as of :pep:`649` and :pep:`695`) provides the necessary scoping semantics for -what is dubbed *class* scope here, in order to support a somewhat similar deferred execution -model for annotations. Of course tag string usage of this scoping is otherwise **not at all** -about annotations. So this PEP separates this concept into two pieces: class scope and then the -machinery specifically supporting annotations. The reference implementation for this PEP only -uses the class scope piece. - -With class scope in place, this means that the lambda wrapping here uses the usual lexical scoping -seen with f-strings. So there's no need to use ``locals()``, ``globals()``, or frame -introspection with ``sys._getframe`` to evaluate the interpolation. Stated differently, -the code of each expression is available and does not have to be looked up with +The class variable ``figure`` here does evaluate correctly with respect to +lexical scope, if at the risk of an HTML injection attack. The reason it +evaluates correctly is that the evaluation is always immediate. First the +expression for ``tag`` is evaluated; then for ``figure``; and finally both +settings are passed into the dynamic construction of the class ``SomeConfig``. + +Because tag strings are supposed to work like f-strings, but with more +capabilities, it's necessary to support annotation scope for the lambda-wrapped +expressions in interpolations. + +But why is annotation scope used, given that this PEP is not at all about +annotations? + +*Annotation* scope (as of :pep:`649` and :pep:`695`) provides the necessary +scoping semantics for names used in class definitions, so as to support a +somewhat similar deferred execution model for annotations. Of course tag string +usage of this scoping is otherwise **not at all** about annotations. So this +PEP, and the corresponding reference implementation, only uses annotation scope, +and not the other machinery specifically supporting annotations. + +With annotation scope in place, lambda-wrapped expressions in interpolations +then provide the usual lexical scoping seen with f-strings. So there's no need +to use ``locals()``, ``globals()``, or frame introspection with +``sys._getframe`` to evaluate the interpolation. In addition, the code of each +expression is available and does not have to be looked up with ``inspect.getsource`` or some other means. Format Specification From 4cab3da7c5ddf4451bd16ce10c673c4a163596eb Mon Sep 17 00:00:00 2001 From: Jim Baker Date: Wed, 17 Jul 2024 22:19:06 -0600 Subject: [PATCH 03/27] Clarify use of annotation scope --- peps/pep-0750.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index 02bc743b87f..0685d4a35a4 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -456,11 +456,11 @@ But why is annotation scope used, given that this PEP is not at all about annotations? *Annotation* scope (as of :pep:`649` and :pep:`695`) provides the necessary -scoping semantics for names used in class definitions, so as to support a -somewhat similar deferred execution model for annotations. Of course tag string -usage of this scoping is otherwise **not at all** about annotations. So this -PEP, and the corresponding reference implementation, only uses annotation scope, -and not the other machinery specifically supporting annotations. +scoping semantics for names used in class definitions, thereby avoiding the +``NameError`` above. In addition, the implementation of these two PEPs provide a +somewhat similar deferred execution model for annotations. However, this PEP and +its reference implementation only use the support for annotation scope; it's up +to the tag function to evaluate any interpolations. With annotation scope in place, lambda-wrapped expressions in interpolations then provide the usual lexical scoping seen with f-strings. So there's no need From 708b47ec0443224dcfcbac78a65b9001c217040f Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Fri, 19 Jul 2024 19:00:16 +0200 Subject: [PATCH 04/27] Mention what happens to named unicodes followed by text --- peps/pep-0750.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index 0685d4a35a4..ff993ddbe92 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -537,6 +537,17 @@ Tag strings will represent this as exactly one ``Decoded`` argument. In this cas '\\N{GRINNING FACE}'.encode('utf-8').decode('unicode-escape') '😀' +Named unicode characters immediately followed by more text will still produce +just one ``Decoded`` argument: + +.. code-block:: python + + def tag(*args): + return args + + assert tag"\N{{GRINNING FACE}}sometext" == ("😀sometext",) + + Return Value ------------ From fb46927176bfc20cb204194b68f118b5eab0887c Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Fri, 19 Jul 2024 19:48:09 +0200 Subject: [PATCH 05/27] Use DecodedConcrete in assertion --- peps/pep-0750.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index ff993ddbe92..64894a35a04 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -545,7 +545,7 @@ just one ``Decoded`` argument: def tag(*args): return args - assert tag"\N{{GRINNING FACE}}sometext" == ("😀sometext",) + assert tag"\N{{GRINNING FACE}}sometext" == (DecodedConcrete("😀sometext"),) Return Value From 7afd44246f95670774fbdc57ba73847c2b66f037 Mon Sep 17 00:00:00 2001 From: Jim Baker Date: Sat, 20 Jul 2024 17:57:08 -0600 Subject: [PATCH 06/27] Rewrite why annotation scope is needed (#4) * Rewrite why annotation scope is needed * Minor copyediting --- peps/pep-0750.rst | 91 +++++++++++++++++++++++++++++------------------ 1 file changed, 57 insertions(+), 34 deletions(-) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index 64894a35a04..a6f4ce0e1d4 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -414,53 +414,69 @@ except that all expressions are always implicitly wrapped with a ``lambda``: lexical scope, including local and global variables. Any valid Python expression can be used, including function and method calls. -However, this PEP further proposes to extend the usual lexical scoping of a -lambda-wrapped expression in the interpolation, as seen with class definitions, -to that supported by `annotation scopes -`_, -so as to minimize developer surprise. +However, there's one additional nuance to consider, `function scope +`_ +versus `annotation scope +`_. +Consider this somewhat contrived example to configure captions: -Let's look at why using annotation scope is important with a somewhat contrived -example where a developer sets up how figures are rendered in a caption, using a -``html`` tag: +.. code-block:: python + + class CaptionConfig: + tag = 'b' + figure = f'<{tag}>Figure' + +Let's now attempt to rewrite the above example to use tag strings: .. code-block:: python - class SomeConfig: + class CaptionConfig: tag = 'b' figure = html'<{tag}>Figure' -Unless the ``html`` function fully evaluates all interpolations before the class -is constructed, a subsequent interpolation will fail with ``NameError: name -'tag' is not defined``. This means that the name ``tag`` is no longer available -in the usual lexical scope for lambdas. Contrast that scoping with what is seen -in using a f-string: +Unfortunately, this rewrite doesn't work if using the usual lambda wrapping to +implement interpolations, namely ``lambda: tag``. When the interpolations are +evaluated by the tag function, it will result in ``NameError: name 'tag' is not +defined``. The root cause of this name error is that ``lambda: tag`` uses function scope, +and it's therefore not able to use the class definition where ``tag`` is +defined. + +Desugaring how the tag string could be evaluated will result in the same +``NameError`` even using f-strings; the lambda wrapping here also uses function +scoping: .. code-block:: python - class SomeConfig: + class CaptionConfig: tag = 'b' - figure = f'<{tag}>Figure' + figure = f'<{(lambda: tag)()}>Figure' -The class variable ``figure`` here does evaluate correctly with respect to -lexical scope, if at the risk of an HTML injection attack. The reason it -evaluates correctly is that the evaluation is always immediate. First the -expression for ``tag`` is evaluated; then for ``figure``; and finally both -settings are passed into the dynamic construction of the class ``SomeConfig``. +For tag strings, getting such a ``NameError`` would be surprising. It would also +be a rough edge in using tag strings in this specific case of working with class +variables. After all, tag strings are supposed to support a superset of the +capabilities of f-strings. -Because tag strings are supposed to work like f-strings, but with more -capabilities, it's necessary to support annotation scope for the lambda-wrapped -expressions in interpolations. +The solution is to use annotation scope for tag string interpolations. While the +name "annotation scope" suggests it's only about annotations, it solves this +problem by lexically resolving names in the class definition, such as ``tag``, +unlike function scope. -But why is annotation scope used, given that this PEP is not at all about -annotations? +.. note:: + + The use of annotation scope means it's not possible to fully desugar + interpolations into Python code. Instead it's as if one is writing + ``interpolation_lambda: tag``, not ``lambda: tag``, where a hypothetical + ``interpolation_lambda`` keyword variant uses annotation scope instead of + the standard function scope. + + This is more or less how the reference implementation implements this + concept (but without creating a new keyword of course). -*Annotation* scope (as of :pep:`649` and :pep:`695`) provides the necessary -scoping semantics for names used in class definitions, thereby avoiding the -``NameError`` above. In addition, the implementation of these two PEPs provide a -somewhat similar deferred execution model for annotations. However, this PEP and -its reference implementation only use the support for annotation scope; it's up -to the tag function to evaluate any interpolations. +This PEP and its reference implementation therefore use the support for +annotation scope. Note that this usage is a separable part from the +implementation of :pep:`649` and :pep:`695` which provides a somewhat similar +deferred execution model for annotations. Instead it's up to the tag function to +evaluate any interpolations. With annotation scope in place, lambda-wrapped expressions in interpolations then provide the usual lexical scoping seen with f-strings. So there's no need @@ -520,7 +536,7 @@ the following, at the cost of losing some type specificity: def mytag(*args: str | tuple) -> Any: ... -A user might write a tag string as a split string: +A user might write a tag string as follows: .. code-block:: python @@ -568,7 +584,14 @@ This is equivalent to: .. code-block:: python - mytag(DecodedConcrete(r'Hi, '), InterpolationConcrete(lambda: name, 'name', 's', 'format_spec'), DecodedConcrete(r'!')) + mytag(DecodedConcrete(r'Hi, '), InterpolationConcrete(lambda: name, 'name', + 's', 'format_spec'), DecodedConcrete(r'!')) + +.. note:: + + To keep it simple, this and subsequent desugaring omits an important scoping + aspect in how names in interpolation expressions are resolved, specifically + when defining classes. See `Interpolation Expression Evaluation`_. No Empty Decoded String ----------------------- From 695ee126aa8487bda2ecb7898426e1122d8984e2 Mon Sep 17 00:00:00 2001 From: David Foster Date: Mon, 8 Jul 2024 22:17:30 -0400 Subject: [PATCH 07/27] PEP 747: Fix rules related to UnionType (T1 | T2). Contrast TypeExpr with TypeAlias. Apply other feedback. (#3856) --- peps/pep-0747.rst | 203 +++++++++++++++++++++++++++++----------------- 1 file changed, 127 insertions(+), 76 deletions(-) diff --git a/peps/pep-0747.rst b/peps/pep-0747.rst index 2bf954c1afe..98bc6f080e5 100644 --- a/peps/pep-0747.rst +++ b/peps/pep-0747.rst @@ -254,9 +254,8 @@ A ``TypeExpr`` value represents a :ref:`type expression such as ``str | None``, ``dict[str, int]``, or ``MyTypedDict``. A ``TypeExpr`` type is written as ``TypeExpr[T]`` where ``T`` is a type or a type variable. It can also be -written without brackets as just ``TypeExpr``, in which case a type -checker should apply its usual type inference mechanisms to determine -the type of its argument, possibly ``Any``. +written without brackets as just ``TypeExpr``, which is treated the same as +to ``TypeExpr[Any]``. Using TypeExprs @@ -278,7 +277,6 @@ or a variable type: :: STR_TYPE: TypeExpr = str # variable type - assert_type(STR_TYPE, TypeExpr[str]) Note however that an *unannotated* variable assigned a type expression literal will not be inferred to be of ``TypeExpr`` type by type checkers because PEP @@ -352,7 +350,7 @@ not spell a type are not ``TypeExpr`` values. :: OPTIONAL_INT_TYPE: TypeExpr = TypeExpr[int | None] # OK - assert isassignable(Optional[int], OPTIONAL_INT_TYPE) + assert isassignable(int | None, OPTIONAL_INT_TYPE) .. _non_universal_typeexpr: @@ -442,14 +440,29 @@ so must be disambiguated based on its argument type: - As a value expression, ``Annotated[x, ...]`` has type ``object`` if ``x`` has a type that is not ``type[C]`` or ``TypeExpr[T]``. -**Union**: The type expression ``T1 | T2`` is ambiguous with the value ``int1 | int2``, -so must be disambiguated based on its argument type: +**Union**: The type expression ``T1 | T2`` is ambiguous with +the value ``int1 | int2``, ``set1 | set2``, ``dict1 | dict2``, and more, +so must be disambiguated based on its argument types: + +- As a value expression, ``x | y`` has type equal to the return type of ``type(x).__or__`` + if ``type(x)`` overrides the ``__or__`` method. + + - When ``x`` has type ``builtins.type``, ``types.GenericAlias``, or the + internal type of a typing special form, ``type(x).__or__`` has a return type + in the format ``TypeExpr[T1 | T2]``. + +- As a value expression, ``x | y`` has type equal to the return type of ``type(y).__ror__`` + if ``type(y)`` overrides the ``__ror__`` method. + + - When ``y`` has type ``builtins.type``, ``types.GenericAlias``, or the + internal type of a typing special form, ``type(y).__ror__`` has a return type + in the format ``TypeExpr[T1 | T2]``. -- As a value expression, ``x | y`` has type ``TypeExpr[x | y]`` - if ``x`` has type ``TypeExpr[t1]`` (or ``type[t1]``) - and ``y`` has type ``TypeExpr[t2]`` (or ``type[t2]``). -- As a value expression, ``x | y`` has type ``int`` - if ``x`` has type ``int`` and ``y`` has type ``int`` +- As a value expression, ``x | y`` has type ``UnionType`` + in all other situations. + + - This rule is intended to be consistent with the preexisting fallback rule + used by static type checkers. The **stringified type expression** ``"T"`` is ambiguous with both the stringified annotation expression ``"T"`` @@ -466,71 +479,24 @@ New kinds of type expressions that are introduced should define how they will be recognized in a value expression context. -Implicit Annotation Expression Values -''''''''''''''''''''''''''''''''''''' - -Although this PEP is mostly concerned with *type expressions* rather than -*annotation expressions*, it is straightforward to extend the rules for -:ref:`recognizing type expressions ` -to similar rules for recognizing annotation expressions, -so this PEP takes the opportunity to define those rules as well: - -The following **unparameterized annotation expressions** can be recognized unambiguously: - -- As a value expression, ``X`` has type ``object``, - for each of the following values of X: - - - ```` - -The following **parameterized annotation expressions** can be recognized unambiguously: - -- As a value expression, ``X`` has type ``object``, - for each of the following values of X: - - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - -**Annotated**: The annotation expression ``Annotated[...]`` is ambiguous with -the type expression ``Annotated[...]``, -so must be :ref:`disambiguated based on its argument type `. - -The following **syntactic annotation expressions** -cannot be recognized in a value expression context at all: - -- ``'*' unpackable`` -- ``name '.' 'args'`` (where ``name`` must be an in-scope ParamSpec) -- ``name '.' 'kwargs'`` (where ``name`` must be an in-scope ParamSpec) - -The **stringified annotation expression** ``"T"`` is ambiguous with both -the stringified type expression ``"T"`` -and the string literal ``"T"``, and -cannot be recognized in a value expression context at all: - -- As a value expression, ``"T"`` continues to have type ``Literal["T"]``. - -No other kinds of annotation expressions currently exist. - -New kinds of annotation expressions that are introduced should define how they -will (or will not) be recognized in a value expression context. - - Literal[] TypeExprs ''''''''''''''''''' -To simplify static type checking, a ``Literal[...]`` value is *not* -considered assignable to a ``TypeExpr`` variable even if all of its members -spell valid types: +A value of ``Literal[...]`` type is *not* considered assignable to +a ``TypeExpr`` variable even if all of its members spell valid types because +dynamic values are not allowed in type expressions: :: STRS_TYPE_NAME: Literal['str', 'list[str]'] = 'str' STRS_TYPE: TypeExpr = STRS_TYPE_NAME # ERROR: Literal[] value is not a TypeExpr +However ``Literal[...]`` itself is still a ``TypeExpr``: + +:: + + DIRECTION_TYPE: TypeExpr[Literal['left', 'right']] = Literal['left', 'right'] # OK + Static vs. Runtime Representations of TypeExprs ''''''''''''''''''''''''''''''''''''''''''''''' @@ -569,6 +535,9 @@ Subtyping Whether a ``TypeExpr`` value can be assigned from one variable to another is determined by the following rules: +Relationship with type +'''''''''''''''''''''' + ``TypeExpr[]`` is covariant in its argument type, just like ``type[]``: - ``TypeExpr[T1]`` is a subtype of ``TypeExpr[T2]`` iff ``T1`` is a @@ -576,12 +545,25 @@ determined by the following rules: - ``type[C1]`` is a subtype of ``TypeExpr[C2]`` iff ``C1`` is a subtype of ``C2``. -A plain ``type`` can be assigned to a plain ``TypeExpr`` but not the -other way around: +An unparameterized ``type`` can be assigned to an unparameterized ``TypeExpr`` +but not the other way around: - ``type[Any]`` is assignable to ``TypeExpr[Any]``. (But not the other way around.) +Relationship with UnionType +''''''''''''''''''''''''''' + +``TypeExpr[U]`` is a subtype of ``UnionType`` iff ``U`` is +the type expression ``X | Y | ...``: + +- ``TypeExpr[X | Y | ...]`` is a subtype of ``UnionType``. + +``UnionType`` is assignable to ``TypeExpr[Any]``. + +Relationship with object +'''''''''''''''''''''''' + ``TypeExpr[]`` is a kind of ``object``, just like ``type[]``: - ``TypeExpr[T]`` for any ``T`` is a subtype of ``object``. @@ -623,11 +605,33 @@ Changed signatures '''''''''''''''''' The following signatures related to type expressions introduce -``TypeExpr`` where previously ``object`` existed: +``TypeExpr`` where previously ``object`` or ``Any`` existed: - ``typing.cast`` - ``typing.assert_type`` +The following signatures transforming union type expressions introduce +``TypeExpr`` where previously ``UnionType`` existed so that a more-precise +``TypeExpr`` type can be inferred: + +- ``builtins.type[T].__or__`` + + - Old: ``def __or__(self, value: Any, /) -> types.UnionType: ...`` + - New: ``def __or__[T2](self, value: TypeExpr[T2], /) -> TypeExpr[T | T2]: ...`` + +- ``builtins.type[T].__ror__`` + + - Old: ``def __ror__(self, value: Any, /) -> types.UnionType: ...`` + - New: ``def __ror__[T1](self, value: TypeExpr[T1], /) -> TypeExpr[T1 | T]: ...`` + +- ``types.GenericAlias.{__or__,__ror__}`` +- «the internal type of a typing special form»``.{__or__,__ror__}`` + +However the implementations of those methods continue to return ``UnionType`` +instances at runtime so that runtime ``isinstance`` checks like +``isinstance('42', int | str)`` and ``isinstance(int | str, UnionType)`` +continue to work. + Unchanged signatures '''''''''''''''''''' @@ -662,12 +666,32 @@ not propose those changes now: - Returns annotation expressions +The following signatures accepting union type expressions continue +to use ``UnionType``: + +- ``builtins.isinstance`` +- ``builtins.issubclass`` +- ``typing.get_origin`` (used in an ``@overload``) + +The following signatures transforming union type expressions continue +to use ``UnionType`` because it is not possible to infer a more-precise +``TypeExpr`` type: + +- ``types.UnionType.{__or__,__ror__}`` + Backwards Compatibility ======================= -Previously the rules for recognizing type expression objects -in a value expression context were not defined, so static type checkers +As a value expression, ``X | Y`` previously had type ``UnionType`` (via :pep:`604`) +but this PEP gives it the more-precise static type ``TypeExpr[X | Y]`` +(a subtype of ``UnionType``) while continuing to return a ``UnionType`` instance at runtime. +Preserving compability with ``UnionType`` is important because ``UnionType`` +supports ``isinstance`` checks, unlike ``TypeExpr``, and existing code relies +on being able to perform those checks. + +The rules for recognizing other kinds of type expression objects +in a value expression context were not previously defined, so static type checkers `varied in what types were assigned `_ to such objects. Existing programs manipulating type expression objects were already limited in manipulating them as plain ``object`` values, @@ -711,12 +735,38 @@ assigned to variables and manipulated like any other data in a program: ``TypeExpr[]`` is how you spell the type of a variable containing a type annotation object describing a type. -``TypeExpr[]`` is similar to ``type[]``, but ``type[]`` can only used to +``TypeExpr[]`` is similar to ``type[]``, but ``type[]`` can only spell simple **class objects** like ``int``, ``str``, ``list``, or ``MyClass``. ``TypeExpr[]`` by contrast can additionally spell more complex types, including those with brackets (like ``list[int]``) or pipes (like ``int | None``), and including special types like ``Any``, ``LiteralString``, or ``Never``. +A ``TypeExpr`` variable looks similar to a ``TypeAlias`` definition, but +can only be used where a dynamic value is expected. +``TypeAlias`` (and the ``type`` statement) by contrast define a name that can +be used where a fixed type is expected: + +- Okay, but discouraged in Python 3.12+: + + :: + + MaybeFloat: TypeAlias = float | None + def sqrt(n: float) -> MaybeFloat: ... + +- Yes: + + :: + + type MaybeFloat = float | None + def sqrt(n: float) -> MaybeFloat: ... + +- No: + + :: + + maybe_float: TypeExpr = float | None + def sqrt(n: float) -> maybe_float: ... # ERROR: Can't use TypeExpr value in a type annotation + It is uncommon for a programmer to define their *own* function which accepts a ``TypeExpr`` parameter or returns a ``TypeExpr`` value. Instead it is more common for a programmer to pass a literal type expression to an *existing* function @@ -891,8 +941,9 @@ The following will be true when `mypy#9773 `__ is implemented: The mypy type checker supports ``TypeExpr`` types. - A reference implementation of the runtime component is provided in the - ``typing_extensions`` module. + +A reference implementation of the runtime component is provided in the +``typing_extensions`` module. Rejected Ideas From 77208880a7f34ec2063097d799e16f9e80a3cbbd Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 10 Jul 2024 14:28:34 -0700 Subject: [PATCH 08/27] PEP 694: Fix typo (#3859) --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 1f54712222f..30e3a32b8bd 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -512,7 +512,7 @@ may have happened on this request. The ``errors`` key is an array of specific errors, each of which contains a ``source`` key, which is a string that indicates what the source of the -error is, and a ``messasge`` key for that specific error. +error is, and a ``message`` key for that specific error. The ``message`` and ``source`` strings do not have any specific meaning, and are intended for human interpretation to figure out what the underlying issue From c77b6ff6faf3aaa531193af2f9ad7abafae61247 Mon Sep 17 00:00:00 2001 From: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Thu, 11 Jul 2024 17:35:42 +0200 Subject: [PATCH 09/27] PEP 2026: Update following discussion (#3860) Co-authored-by: Erlend E. Aasland --- peps/pep-2026.rst | 270 +++++++++++++++++++++++++++++++++++++--------- 1 file changed, 217 insertions(+), 53 deletions(-) diff --git a/peps/pep-2026.rst b/peps/pep-2026.rst index e297dca9430..913e062d1d3 100644 --- a/peps/pep-2026.rst +++ b/peps/pep-2026.rst @@ -14,9 +14,18 @@ Abstract This PEP proposes updating the versioning scheme for Python to include the calendar year. -This aims to make the support lifecycle clear -by making it easy to see when a version was first released, -and easier to work out when it will reach end of life (EOL). + +Calendar Versioning (CalVer) makes *everything* easier to translate into +calendar time rather than counting versions and looking up when they will be +(or were) released: + +* The support lifecycle is clear, + making it easy to see when a version was first released. +* Deprecations are easier to manage for maintainers and users. +* It's easier to work out when a version will reach end of life (EOL). +* It helps people, especially new learners, understand how old their installation is. +* It's easier to reason about which versions of Python to support + for libraries and applications. Starting with what would have been Python 3.15, the version is 3.YY.micro where YY is the year of initial release: @@ -201,6 +210,21 @@ and encode the year in the minor version: For example, 3.26 will be released in 2026. It makes it obvious when a release first came out. +Clarity of deprecation removal +------------------------------ + +Warnings for deprecations often mention the version they will be removed in. +For example: + + DeprecationWarning: 'ctypes.SetPointerType' is deprecated and slated for + removal in Python 3.15 + +However, once aware of CalVer, it is immediately obvious from the warning how +long you have to to take action: + + DeprecationWarning: 'ctypes.SetPointerType' is deprecated and slated for + removal in Python 3.26 + Clarity of support lifecycle ---------------------------- @@ -219,6 +243,45 @@ it’s much easier: "26 + 5 = [20]31" +Clarity of installation age +--------------------------- + +With the year in the version, it’s easier to work out how old your installation +is. For example, with the current scheme, if you're using Python 3.15 in 2035, +it's not immediately clear that it was first released in 2026 (and has been EOL +since 2031). + +With knowledge of CalVer, if you're using Python 3.26 in 2035, it's clear it was +first released nine years ago and it's probably time to upgrade. + +This can help prompt people to switch to supported releases still under security +support, and help in teaching new users who may have older installations. + +Clarity of version support +-------------------------- + +CalVer makes it easier to reason about which versions of Python to support. + +For example, without CalVer, setting your minimum compatible Python version to +3.19 in 2031 sets an aggressive assumption regarding version adoption and +support. + +However, with CalVer, this is more obvious if setting the minimum to 3.30 in +2031. For wider support, perhaps you prefer setting it to 3.26. + +Similarly, library maintainers supporting all CPython upstream versions +need to test against five versions (or six including the pre-release). + +For example, in 2030, the supported versions without CalVer would be: + +* 3.15, 3.16, 3.17, 3.18, 3.19 + +With CalVer they would be: + +* 3.26, 3.27, 3.28, 3.29, 3.30 + +A maintainer can see at a glance which versions are current and need testing. + Non-goals --------- @@ -280,56 +343,6 @@ Python 3.14 will be the last version before this change, released in 2025. Python 3.26 will be the first version after this change, released in 2026. There will be no Python 3.15 to 3.25 inclusive. -Backwards compatibility -======================= - -This version change is the safest of the CalVer options considered -(see `rejected ideas `_): we keep 3 as the major version, -and the minor version is still two digits. -The minor will eventually change to three digits but this is predictable, -a long way off and can be planned for. - -We retain the ``python3`` executable. - -Version mapping ---------------- - -Versions 3.15 to 3.25 inclusive will be skipped. -Features, deprecations and removals planned for these will be remapped to the -new version numbers. - -For example, a deprecation initially planned for removal in 3.16 will instead -be removed in 3.27. - -+-------------+------------------+-----------------+ -| Old version | New version | Initial release | -+=============+==================+=================+ -| 3.14 | 3.14 (no change) | 2025 | -+-------------+------------------+-----------------+ -| 3.15 | 3.26 | 2026 | -+-------------+------------------+-----------------+ -| 3.16 | 3.27 | 2027 | -+-------------+------------------+-----------------+ -| 3.17 | 3.28 | 2028 | -+-------------+------------------+-----------------+ -| 3.18 | 3.29 | 2029 | -+-------------+------------------+-----------------+ -| 3.19 | 3.30 | 2030 | -+-------------+------------------+-----------------+ -| 3.20 | 3.31 | 2031 | -+-------------+------------------+-----------------+ -| 3.21 | 3.32 | 2032 | -+-------------+------------------+-----------------+ -| 3.22 | 3.33 | 2033 | -+-------------+------------------+-----------------+ -| 3.23 | 3.34 | 2034 | -+-------------+------------------+-----------------+ -| 3.24 | 3.35 | 2035 | -+-------------+------------------+-----------------+ -| 3.25 | 3.36 | 2036 | -+-------------+------------------+-----------------+ - - Security implications ===================== @@ -359,6 +372,8 @@ errors out and tells the user to use ``python3.26`` instead. Rejected ideas ============== +.. _PEP 2026 YY.0: + YY.0 ---- @@ -381,6 +396,8 @@ Will we stick with `version 3 forever Another option would be to put the year in the major version and jump to 26.0. This could mean we could leapfrog all that 4.0 baggage. +.. _PEP 2026 Platform compatibility tags: + Platform compatibility tags ''''''''''''''''''''''''''' @@ -466,6 +483,8 @@ it's still a 3 followed by two digits. `Flake8's flake8-2020 plugin `__ to help find the problems like these. +.. _PEP 2026 python3 command: + ``python3`` command ''''''''''''''''''' @@ -547,6 +566,151 @@ Change during 3.14 cycle The Python 3.14 release must go ahead because: π. +Backwards compatibility +======================= + +This version change is the safest of the CalVer options considered +(see `rejected ideas `_): we keep 3 as the major version, +and the minor version is still two digits. +The minor will eventually change to three digits but this is predictable, +a long way off and can be planned for. + +We retain the ``python3`` executable. + +Version mapping +--------------- + +Versions 3.15 to 3.25 inclusive will be skipped. +Features, deprecations and removals planned for these will be remapped to the +new version numbers. + +For example, a deprecation initially planned for removal in 3.16 will instead +be removed in 3.27. + ++-------------+------------------+-----------------+ +| Old version | New version | Initial release | ++=============+==================+=================+ +| 3.14 | 3.14 (no change) | 2025 | ++-------------+------------------+-----------------+ +| 3.15 | 3.26 | 2026 | ++-------------+------------------+-----------------+ +| 3.16 | 3.27 | 2027 | ++-------------+------------------+-----------------+ +| 3.17 | 3.28 | 2028 | ++-------------+------------------+-----------------+ +| 3.18 | 3.29 | 2029 | ++-------------+------------------+-----------------+ +| 3.19 | 3.30 | 2030 | ++-------------+------------------+-----------------+ +| 3.20 | 3.31 | 2031 | ++-------------+------------------+-----------------+ +| 3.21 | 3.32 | 2032 | ++-------------+------------------+-----------------+ +| 3.22 | 3.33 | 2033 | ++-------------+------------------+-----------------+ +| 3.23 | 3.34 | 2034 | ++-------------+------------------+-----------------+ +| 3.24 | 3.35 | 2035 | ++-------------+------------------+-----------------+ +| 3.25 | 3.36 | 2036 | ++-------------+------------------+-----------------+ + +Forwards compatibility +====================== + +Future change in cadence +------------------------ + +This PEP proposes no change to the annual release cadence as defined in +:pep:`602`, which lays out +:pep:`many good reasons for annual releases <602#rationale-and-goals>` +(for example, smaller releases with a predictable release calendar, +and syncing with external redistributors). +However unlikely, should we decide to change the cadence in the future, CalVer +does not preclude doing so. + +Less frequent +''''''''''''' + +If we went to *fewer than one release per year*, the proposed CalVer scheme +still works; indeed, it even helps people know in which year to expect the +release. For example, if we released every second year starting in 2036: + +* 3.36.0 would be released in 2036 +* 3.38.0 would be released in 2038 +* and so on + +Ecosystem changes depend in part on how the the hypothetical cadence-changing +PEP updates :pep:`387` (Backwards Compatibility Policy). If, for example, it +requires that the deprecation period must be at least one feature release and +not the current two (to maintain the minimum two years), CalVer has the benefit +over the status quo in requiring no changes to planned removal versions +(other than adjusting any falling in non-release years). + +.. _PEP 2026 More frequent: + +More frequent +''''''''''''' + +If we went to *more than one release per year*, here are some options. +For example, if we released in April and October starting in 2036, the next +four releases could be: + ++---------------+--------------------------------+-----------+-----------+-----------+-----------+ +| Scheme | Notes | 2036 a | 2036 b | 2037 a | 2037 b | ++===============+================================+===========+===========+===========+===========+ +| YY.MM.micro | Year as major, month as minor | 36.04.0 | 36.10.0 | 37.04.0 | 37.10.0 | ++---------------+--------------------------------+-----------+-----------+-----------+-----------+ +| YY.x.micro | Year as major, | 36.1.0 | 36.2.0 | 37.1.0 | 37.2.0 | +| | serial number as minor | | | | | ++---------------+--------------------------------+-----------+-----------+-----------+-----------+ +| 3.YYMM.micro | Combine year and month | 3.3604.0 | 3.3610.0 | 3.3704.0 | 3.3710.0 | +| | as minor | | | | | ++---------------+--------------------------------+-----------+-----------+-----------+-----------+ +| 3.YYx.micro | Combine year and serial number | 3.360.0 | 3.361.0 | 3.370.0 | 3.371.0 | +| | as minor | | | | | ++---------------+--------------------------------+-----------+-----------+-----------+-----------+ +| 3.YY.MM.micro | Add an extra month segment | 3.36.04.0 | 3.36.10.0 | 3.37.04.0 | 3.37.10.0 | ++---------------+--------------------------------+-----------+-----------+-----------+-----------+ +| 3.major.micro | No more CalVer: | 3.36.0 | 3.37.0 | 3.38.0 | 3.39.0 | +| | increment minor +-----------+-----------+-----------+-----------+ +| | | 3.50.0 | 3.51.0 | 3.52.0 | 3.53.0 | +| | +-----------+-----------+-----------+-----------+ +| | | 3.100.0 | 3.101.0 | 3.102.0 | 3.103.0 | ++---------------+--------------------------------+-----------+-----------+-----------+-----------+ +| 4.major.micro | No more CalVer: | 4.0.0 | 4.1.0 | 4.2.0 | 4.3.0 | ++---------------+ increment major +-----------+-----------+-----------+-----------+ +| 5.major.micro | | 5.0.0 | 5.1.0 | 5.2.0 | 5.3.0 | ++---------------+--------------------------------+-----------+-----------+-----------+-----------+ + +The YY options would require addressing issues around the +`platform compatibility tags `__, +the `python3 command `_, and code +`assuming the version always begins with 3 `__. + +The options keeping major version 3 but changing the minor to three or four +digits would also need to address code +`assuming the version is always two digits `__. + +The option adding an extra month segment is the biggest change as code would +need to deal with a four-part version instead of three. + +The options dropping CalVer would be the most conservative +allowing the major and minor to be chosen freely. + +No more CalVer +-------------- + +Adopting CalVer now does not preclude moving away CalVer in the future, +for example, back to the original scheme, to SemVer or another scheme. +Some options are `listed in the table above `__. +If wanting to make it clear the minor is no longer the year, +it can be bumped to a higher round number (for example, 3.50 or 3.100) +or the major version can be bumped (for example, to 4.0 or 5.0). +Additionally, a `version epoch +`__ +could be considered. + Footnotes ========= From fc0b9592d18476db00cb47408e94028af2f30699 Mon Sep 17 00:00:00 2001 From: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Mon, 15 Jul 2024 11:09:45 +0200 Subject: [PATCH 10/27] PEP 101: Remove outdated info and add new info (#3863) * PEP 101: Remove outdated info * PEP 101: Update make command for running tests * PEP 101: Replace '#python-dev and/or python-committers' with 'Discord and/or Discourse * PEP 101: Add Hugo as 3.14 RM * PEP 101: Add to PSRT --- .github/CODEOWNERS | 2 +- peps/pep-0101.rst | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 6c466af74f0..45741d03c2c 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -50,7 +50,7 @@ peps/pep-0020.rst @tim-one peps/pep-0042.rst @jeremyhylton # ... peps/pep-0100.rst @malemburg -peps/pep-0101.rst @Yhg1s @pablogsal @ambv @ned-deily +peps/pep-0101.rst @hugovk @Yhg1s @pablogsal @ambv @ned-deily peps/pep-0102.rst @warsaw @gvanrossum # peps/pep-0103.rst # ... diff --git a/peps/pep-0101.rst b/peps/pep-0101.rst index 1da2bfa0b28..d36ec2e2bbd 100644 --- a/peps/pep-0101.rst +++ b/peps/pep-0101.rst @@ -50,9 +50,6 @@ Here's a hopefully-complete list. `Misc/NEWS `_ management tool. You can pip install it. - * A fairly complete installation of a recent TeX distribution, - such as texlive. You need that for building the PDF docs. - * Access to servers where you will upload files: * ``downloads.nyc1.psf.io``, the server that hosts download files; and @@ -80,6 +77,9 @@ Here's a hopefully-complete list. account, or a redirecting alias + SMTP credentials to send email from this address that looks legit to major email providers. +* Be added to the `Python Security Response Team + `__. + Types of Releases ================= @@ -124,6 +124,7 @@ release. The roles and their current experts are: * RM = Release Manager + - Hugo van Kemenade (FI) - Thomas Wouters (NL) - Pablo Galindo Salgado (UK) - Ɓukasz Langa (PL) @@ -331,7 +332,7 @@ to perform some manual editing steps. If you're feeling lucky and have some time to kill, or if you are making a release candidate or **final** release, run the full test suite:: - make testall + make buildbottest If the tests pass, then you can feel good that the tarball is fine. If some of the tests fail, or anything else about the @@ -388,8 +389,8 @@ to perform some manual editing steps. to your home directory on ``downloads.nyc1.psf.io``. While you're waiting for the files to finish uploading, you can continue - on with the remaining tasks. You can also ask folks on #python-dev - and/or python-committers to download the files as they finish uploading + on with the remaining tasks. You can also ask folks on Discord + and/or `discuss.python.org`_ to download the files as they finish uploading so that they can test them on their platforms as well. - Now you need to go to ``downloads.nyc1.psf.io`` and move all the files in place @@ -624,8 +625,7 @@ Now it's time to twiddle the website. Almost none of this is automated, sorry. To do these steps, you must have the permission to edit the website. If you don't have that, ask someone on pydotorg@python.org for the proper -permissions. (Or ask Ewa, who coordinated the effort for the new website -with RevSys.) +permissions. - Log in to https://www.python.org/admin From 27334b13b063f8cd6328896b1d66405c9d5efdab Mon Sep 17 00:00:00 2001 From: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Tue, 16 Jul 2024 21:26:06 +0300 Subject: [PATCH 11/27] PEP 11: Add Russell as an iOS contact (#3865) --- peps/pep-0011.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0011.rst b/peps/pep-0011.rst index f51a8562fc2..d8f10329cf7 100644 --- a/peps/pep-0011.rst +++ b/peps/pep-0011.rst @@ -122,8 +122,8 @@ Tier 3 Target Triple Notes Contacts ================================ =========================== ======== aarch64-pc-windows-msvc Steve Dower -arm64-apple-ios iOS on device Ned Deily -arm64-apple-ios-simulator iOS on M1 macOS simulator Ned Deily +arm64-apple-ios iOS on device Russell Keith-Magee, Ned Deily +arm64-apple-ios-simulator iOS on M1 macOS simulator Russell Keith-Magee, Ned Deily armv7l-unknown-linux-gnueabihf Raspberry Pi OS, glibc, gcc Gregory P. Smith powerpc64le-unknown-linux-gnu glibc, clang Victor Stinner From 40226661594e0ce57e41cd49fc9b8784802f6a87 Mon Sep 17 00:00:00 2001 From: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Wed, 17 Jul 2024 08:04:03 +0300 Subject: [PATCH 12/27] Meta: Document the PEPs API (#3864) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- .pre-commit-config.yaml | 10 +- .../pep_zero_generator/writer.py | 7 ++ peps/api/index.rst | 91 +++++++++++++++++++ peps/conf.py | 24 +++-- peps/contents.rst | 1 + 5 files changed, 118 insertions(+), 15 deletions(-) create mode 100644 peps/api/index.rst diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 9ebe7b8dcf8..da6e98e9755 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -12,7 +12,7 @@ default_stages: [commit] repos: # General file checks and fixers - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v4.5.0 + rev: v4.6.0 hooks: - id: mixed-line-ending name: "Normalize mixed line endings" @@ -43,17 +43,17 @@ repos: name: "Check YAML" - repo: https://github.com/psf/black-pre-commit-mirror - rev: 24.1.1 + rev: 24.4.2 hooks: - id: black name: "Format with Black" args: - '--target-version=py39' - '--target-version=py310' - files: 'pep_sphinx_extensions/tests/.*' + files: '^(peps/conf\.py|pep_sphinx_extensions/tests/.*)$' - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.2.1 + rev: v0.5.1 hooks: - id: ruff name: "Lint with Ruff" @@ -89,7 +89,7 @@ repos: # Manual codespell check - repo: https://github.com/codespell-project/codespell - rev: v2.2.6 + rev: v2.3.0 hooks: - id: codespell name: "Check for common misspellings in text files" diff --git a/pep_sphinx_extensions/pep_zero_generator/writer.py b/pep_sphinx_extensions/pep_zero_generator/writer.py index f8232d0ceb0..e0507cc7f29 100644 --- a/pep_sphinx_extensions/pep_zero_generator/writer.py +++ b/pep_sphinx_extensions/pep_zero_generator/writer.py @@ -169,6 +169,13 @@ def write_pep0( self.emit_newline() self.emit_newline() + self.emit_title("API") + self.emit_text( + "The `PEPS API `__ is a JSON file of metadata about " + "all the published PEPs. :doc:`Read more here `." + ) + self.emit_newline() + # PEPs by category self.emit_title("Index by Category") meta, info, provisional, accepted, open_, finished, historical, deferred, dead = _classify_peps(peps) diff --git a/peps/api/index.rst b/peps/api/index.rst new file mode 100644 index 00000000000..46516f4150b --- /dev/null +++ b/peps/api/index.rst @@ -0,0 +1,91 @@ +PEPs API +======== + +There is a read-only API of published PEPs available at: + +* https://peps.python.org/api/peps.json + +The structure is like: + +.. code-block:: javascript + + { + "": { + "number": integer, + "title": string, + "authors": string, + "discussions_to": string | null, + "status": "Accepted" | "Active" | "Deferred" | "Draft" | "Final" | "Provisional" | "Rejected" | "Superseded" | "Withdrawn", + "type": "Informational" | "Process" | "Standards Track", + "topic": "governance" | "packaging" | "release" | "typing" | "", + "created": string, + "python_version": string | null, + "post_history": string, + "resolution": string | null, + "requires": string | null, + "replaces": string | null, + "superseded_by": string | null, + "url": string + }, + } + +Date values are formatted as DD-MMM-YYYY, +and multiple dates are combined in a comma-separated list. + +For example: + +.. code-block:: json + + { + "8": { + "number": 8, + "title": "Style Guide for Python Code", + "authors": "Guido van Rossum, Barry Warsaw, Alyssa Coghlan", + "discussions_to": null, + "status": "Active", + "type": "Process", + "topic": "", + "created": "05-Jul-2001", + "python_version": null, + "post_history": "05-Jul-2001, 01-Aug-2013", + "resolution": null, + "requires": null, + "replaces": null, + "superseded_by": null, + "url": "https://peps.python.org/pep-0008/" + }, + "484": { + "number": 484, + "title": "Type Hints", + "authors": "Guido van Rossum, Jukka Lehtosalo, Ɓukasz Langa", + "discussions_to": "python-dev@python.org", + "status": "Final", + "type": "Standards Track", + "topic": "typing", + "created": "29-Sep-2014", + "python_version": "3.5", + "post_history": "16-Jan-2015, 20-Mar-2015, 17-Apr-2015, 20-May-2015, 22-May-2015", + "resolution": "https://mail.python.org/pipermail/python-dev/2015-May/140104.html", + "requires": null, + "replaces": null, + "superseded_by": null, + "url": "https://peps.python.org/pep-0484/" + }, + "622": { + "number": 622, + "title": "Structural Pattern Matching", + "authors": "Brandt Bucher, Daniel F Moisset, Tobias Kohn, Ivan Levkivskyi, Guido van Rossum, Talin", + "discussions_to": "python-dev@python.org", + "status": "Superseded", + "type": "Standards Track", + "topic": "", + "created": "23-Jun-2020", + "python_version": "3.10", + "post_history": "23-Jun-2020, 08-Jul-2020", + "resolution": null, + "requires": null, + "replaces": null, + "superseded_by": "634", + "url": "https://peps.python.org/pep-0622/" + } + } diff --git a/peps/conf.py b/peps/conf.py index d23763b3430..9a747aa8fa1 100644 --- a/peps/conf.py +++ b/peps/conf.py @@ -30,7 +30,7 @@ ".rst": "pep", } -# List of patterns (relative to source dir) to ignore when looking for source files. +# List of patterns (relative to source dir) to include when looking for source files. include_patterns = [ # Required for Sphinx "contents.rst", @@ -38,9 +38,12 @@ "pep-????.rst", # PEP ancillary files "pep-????/*.rst", + # PEPs API + "api/*.rst", # Documentation "docs/*.rst", ] +# And to ignore when looking for source files. exclude_patterns = [ # PEP Template "pep-0012/pep-NNNN.rst", @@ -51,14 +54,14 @@ # Intersphinx configuration intersphinx_mapping = { - 'python': ('https://docs.python.org/3/', None), - 'packaging': ('https://packaging.python.org/en/latest/', None), - 'typing': ('https://typing.readthedocs.io/en/latest/', None), - 'trio': ('https://trio.readthedocs.io/en/latest/', None), - 'devguide': ('https://devguide.python.org/', None), - 'py3.11': ('https://docs.python.org/3.11/', None), - 'py3.12': ('https://docs.python.org/3.12/', None), - 'py3.13': ('https://docs.python.org/3.13/', None), + "python": ("https://docs.python.org/3/", None), + "packaging": ("https://packaging.python.org/en/latest/", None), + "typing": ("https://typing.readthedocs.io/en/latest/", None), + "trio": ("https://trio.readthedocs.io/en/latest/", None), + "devguide": ("https://devguide.python.org/", None), + "py3.11": ("https://docs.python.org/3.11/", None), + "py3.12": ("https://docs.python.org/3.12/", None), + "py3.13": ("https://docs.python.org/3.13/", None), } intersphinx_disabled_reftypes = [] @@ -86,4 +89,5 @@ html_baseurl = "https://peps.python.org" # to create the CNAME file gettext_auto_build = False # speed-ups -templates_path = [os.fspath(_PSE_PATH / "pep_theme" / "templates")] # Theme template relative paths from `confdir` +# Theme template relative paths from `confdir` +templates_path = [os.fspath(_PSE_PATH / "pep_theme" / "templates")] diff --git a/peps/contents.rst b/peps/contents.rst index d791f08f809..5d5c23ee3a6 100644 --- a/peps/contents.rst +++ b/peps/contents.rst @@ -15,4 +15,5 @@ This is an internal Sphinx page; please go to the :doc:`PEP Index `. :caption: PEP Table of Contents (needed for Sphinx): pep-* + api/* topic/* From e8e2c777e5f612b5e19ebf8e5dcfb7345e6074eb Mon Sep 17 00:00:00 2001 From: "T. Wouters" Date: Thu, 18 Jul 2024 16:39:56 +0200 Subject: [PATCH 13/27] PEP 719: Update for today's release of 3.13.0b4 (#3868) --- peps/pep-0719.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0719.rst b/peps/pep-0719.rst index ffb87af9575..0118adcb873 100644 --- a/peps/pep-0719.rst +++ b/peps/pep-0719.rst @@ -46,10 +46,10 @@ Actual: (No new features beyond this point.) - 3.13.0 beta 2: Wednesday, 2024-06-05 - 3.13.0 beta 3: Thursday, 2024-06-27 +- 3.13.0 beta 4: Thursday, 2024-07-18 Expected: -- 3.13.0 beta 4: Tuesday, 2024-07-16 - 3.13.0 candidate 1: Tuesday, 2024-07-30 - 3.13.0 candidate 2: Tuesday, 2024-09-03 - 3.13.0 final: Tuesday, 2024-10-01 From 7368a3dba113015242001ad9c6720016c8b83fbd Mon Sep 17 00:00:00 2001 From: William Woodruff Date: Thu, 18 Jul 2024 11:49:15 -0400 Subject: [PATCH 14/27] PEP 740: Mark as Provisional (#3848) Signed-off-by: William Woodruff --- peps/pep-0740.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/peps/pep-0740.rst b/peps/pep-0740.rst index 28871fc874c..c3945c61c8f 100644 --- a/peps/pep-0740.rst +++ b/peps/pep-0740.rst @@ -6,12 +6,14 @@ Author: William Woodruff , Sponsor: Donald Stufft PEP-Delegate: Donald Stufft Discussions-To: https://discuss.python.org/t/pep-740-index-support-for-digital-attestations/44498 -Status: Draft -Type: Informational +Status: Provisional +Type: Standards Track Topic: Packaging Created: 08-Jan-2024 Post-History: `02-Jan-2024 `__, `29-Jan-2024 `__ +Resolution: https://discuss.python.org/t/pep-740-index-support-for-digital-attestations/44498/26 + Abstract ======== From e248a85adbcbc9996087f9134f421758f3481598 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Tue, 23 Jul 2024 13:49:18 -0700 Subject: [PATCH 15/27] PEP 749: Add section on metaclasses (#3847) Co-authored-by: Carl Meyer --- peps/pep-0749.rst | 165 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 164 insertions(+), 1 deletion(-) diff --git a/peps/pep-0749.rst b/peps/pep-0749.rst index 7b76fc6d867..6503ef832f1 100644 --- a/peps/pep-0749.rst +++ b/peps/pep-0749.rst @@ -187,6 +187,10 @@ The module will contain the following functionality: module, or class. This will replace :py:func:`inspect.get_annotations`. The latter will delegate to the new function. It may eventually be deprecated, but to minimize disruption, we do not propose an immediate deprecation. +* ``get_annotate_function()``: A function that returns the ``__annotate__`` function + of an object, if it has one, or ``None`` if it does not. This is usually equivalent + to accessing the ``.__annotate__`` attribute, except in the presence of metaclasses + (see :ref:`below `). * ``Format``: an enum that contains the possible formats of annotations. This will replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`. PEP 649 proposed to make these values global members of the :py:mod:`inspect` @@ -390,6 +394,163 @@ More specifically: ``__dict__``. Writing to these attributes will directly update the ``__dict__``, without affecting the wrapped callable. +.. _pep749-metaclasses: + +Annotations and metaclasses +=========================== + +Testing of the initial implementation of this PEP revealed serious problems with +the interaction between metaclasses and class annotations. + +Pre-existing bugs +----------------- + +We found several bugs in the existing behavior of ``__annotations__`` on classes +while investigating the behaviors to be specified in this PEP. Fixing these bugs +on Python 3.13 and earlier is outside the scope of this PEP, but they are noted here +to explain the corner cases that need to be dealt with. + +For context, on Python 3.10 through 3.13 the ``__annotations__`` dictionary is +placed in the class namespace if the class has any annotations. If it does not, +there is no ``__annotations__`` class dictionary key when the class is created, +but accessing ``cls.__annotations__`` invokes a descriptor defined on ``type`` +that returns an empty dictionary and stores it in the class dictionary. +:py:ref:`Static types ` are an exception: they never have +annotations, and accessing ``.__annotations__`` raises :py:exc:`AttributeError`. +On Python 3.9 and earlier, the behavior was different; see +`gh-88067 `__. + +The following code fails identically on Python 3.10 through 3.13:: + + class Meta(type): pass + + class X(metaclass=Meta): + a: str + + class Y(X): pass + + Meta.__annotations__ # important + assert Y.__annotations__ == {}, Y.__annotations__ # fails: {'a': } + +If the annotations on the metaclass ``Meta`` are accessed before the annotations +on ``Y``, then the annotations for the base class ``X`` are leaked to ``Y``. +However, if the metaclass's annotations are *not* accessed (i.e., the line ``Meta.__annotations__`` +above is removed), then the annotations for ``Y`` are correctly empty. + +Similarly, annotations from annotated metaclasses leak to unannotated +classes that are instances of the metaclass:: + + class Meta(type): + a: str + + class X(metaclass=Meta): + pass + + assert X.__annotations__ == {}, X.__annotations__ # fails: {'a': } + +The reason for these behaviors is that if the metaclass contains an +``__annotations__`` entry in its class dictionary, this prevents +instances of the metaclass from using the ``__annotations__`` data descriptor +on the base :py:class:`type` class. In the first case, accessing ``Meta.__annotations__`` +sets ``Meta.__dict__["__annotations__"] = {}`` as a side effect. Then, looking +up the ``__annotations__`` attribute on ``Y`` first sees the metaclass attribute, +but skips it because it is a data descriptor. Next, it looks in the class dictionaries +of the classes in its method resolution order (MRO), finds ``X.__annotations__``, +and returns it. In the second example, there are no annotations +anywhere in the MRO, so ``type.__getattribute__`` falls back to +returning the metaclass attribute. + +Metaclass behavior with PEP 649 +------------------------------- + +With :pep:`649`, the behavior of accessing the ``.__annotations__`` attribute +on classes when metaclasses are involved becomes even more erratic, because now +``__annotations__`` is only lazily added to the class dictionary even for classes +with annotations. The new ``__annotate__`` attribute is also lazily created +on classes without annotations, which causes further misbehaviors when +metaclasses are involved. + +The cause of these problems is that we set the ``__annotate__`` and ``__annotations__`` +class dictionary entries only under some circumstances, and rely on descriptors +defined on :py:class:`type` to fill them in if they are not set. When normal +attribute lookup is used, this approach breaks down in the presence of +metaclasses, because entries in the metaclass's own class dictionary can render +the descriptors invisible. + +While we considered several approaches that would allow ``cls.__annotations__`` +and ``cls.__annotate__`` to work reliably when ``cls`` is a type with a custom +metaclass, any such approach would expose significant complexity to advanced users. +Instead, we recommend a simpler approach that confines the complexity to the +``annotationlib`` module: in ``annotationlib.get_annotations``, we bypass normal +attribute lookup by using the ``type.__annotations__`` descriptor directly. + +Specification +------------- + +Users should always use ``annotationlib.get_annotations`` to access the +annotations of a class object, and ``annotationlib.get_annotate_function`` +to access the ``__annotate__`` function. These functions will return only +the class's own annotations, even when metaclasses are involved. + +The behavior of accessing the ``__annotations__`` and ``__annotate__`` +attributes on classes with a metaclass other than ``builtins.type`` is +unspecified. The documentation should warn against direct use of these +attributes and recommend using the ``annotationlib`` module instead. + +Similarly, the presence of ``__annotations__`` and ``__annotate__`` keys +in the class dictionary is an implementation detail and should not be relied +upon. + +Rejected alternatives +--------------------- + +We considered two broad approaches for dealing with the behavior +of the ``__annotations__`` and ``__annotate__`` entries in classes: + +* Ensure that the entry is *always* present in the class dictionary, even if it + is empty or has not yet been evaluated. This means we do not have to rely on + the descriptors defined on :py:class:`type` to fill in the field, and + therefore the metaclass's attributes will not interfere. (Prototype + in `gh-120719 `__.) +* Ensure that the entry is *never* present in the class dictionary, or at least + never added by logic in the language core. This means that the descriptors + on :py:class:`type` will always be used, without interference from the metaclass. + (Prototype in `gh-120816 `__.) + +Alex Waygood suggested an implementation using the first approach. When a +heap type (such as a class created through the ``class`` statement) is created, +``cls.__dict__["__annotations__"]`` is set to a special descriptor. +On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__`` +and returning the result. The annotations dictionary is cached within the +descriptor instance. The descriptor also behaves like a mapping, +so that code that uses ``cls.__dict__["__annotations__"]`` will still usually +work: treating the object as a mapping will evaluate the annotations and behave +as if the descriptor itself was the annotations dictionary. (Code that assumes +that ``cls.__dict__["__annotations__"]`` is specifically an instance of ``dict`` +may break, however.) + +This approach is also straightforward to implement for ``__annotate__``: this +attribute is already always set for classes with annotations, and we can set +it explicitly to ``None`` for classes without annotations. + +While this approach would fix the known edge cases with metaclasses, it +introduces significant complexity to all classes, including a new built-in type +(for the annotations descriptor) with unusual behavior. + +The alternative approach would be to never set ``__dict__["__annotations__"]`` +and use some other storage to store the cached annotations. This behavior +change would have to apply even to classes defined under +``from __future__ import annotations``, because otherwise there could be buggy +behavior if a class is defined without ``from __future__ import annotations`` +but its metaclass does have the future enabled. As :pep:`649` previously noted, +removing ``__annotations__`` from class dictionaries also has backwards compatibility +implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to +retrieve annotations. + +This approach would also mean that accessing ``.__annotations__`` on an instance +of an annotated class no longer works. While this behavior is not documented, +it is a long-standing feature of Python and is relied upon by some users. + Remove code flag for marking ``__annotate__`` functions ======================================================= @@ -695,7 +856,9 @@ Acknowledgments First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his initial decisions, but the overall design is still his. -I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. +I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. Alex Waygood, +Alyssa Coghlan, and David Ellis provided insightful feedback and suggestions on the +interaction between metaclasses and ``__annotations__``. Appendix ======== From be27c4a4123fae4316ee9b5c9893fbc7dfd21dff Mon Sep 17 00:00:00 2001 From: Lavrentiy Rubtsov Date: Thu, 25 Jul 2024 00:02:17 +0500 Subject: [PATCH 16/27] PEP 8: Update a Wikipedia link (#3552) --- peps/pep-0008.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0008.rst b/peps/pep-0008.rst index fa0be88dabc..783093c98ce 100644 --- a/peps/pep-0008.rst +++ b/peps/pep-0008.rst @@ -1631,7 +1631,7 @@ References .. [3] Donald Knuth's *The TeXBook*, pages 195 and 196. -.. [4] http://www.wikipedia.com/wiki/CamelCase +.. [4] http://www.wikipedia.com/wiki/Camel_case .. [5] Typeshed repo https://github.com/python/typeshed From ef06ba17e28924b02d2b562f36b5fa10224ee3d2 Mon Sep 17 00:00:00 2001 From: Mariatta Date: Wed, 24 Jul 2024 15:56:04 -0700 Subject: [PATCH 17/27] PEP 635: Minor typo fix in code sample (#3871) Looks like an unclosed f-string. --- peps/pep-0635.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0635.rst b/peps/pep-0635.rst index 9e25e8f2088..114972a97c7 100644 --- a/peps/pep-0635.rst +++ b/peps/pep-0635.rst @@ -1118,7 +1118,7 @@ be emulated by a user-defined class as follows:: The proposal was to combine patterns with type annotations:: match x: - case [a: int, b: str]: print(f"An int {a} and a string {b}:) + case [a: int, b: str]: print(f"An int {a} and a string {b}:") case [a: int, b: int, c: int]: print("Three ints", a, b, c) ... From 482f22482f5605b3e47818062310873a933ea0ad Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 16:13:38 -0700 Subject: [PATCH 18/27] PEP 751: A file format to list Python dependencies for installation reproducibility (#3870) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Jelle Zijlstra Co-authored-by: Carol Willing --- peps/pep-0751.rst | 934 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 934 insertions(+) create mode 100644 peps/pep-0751.rst diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst new file mode 100644 index 00000000000..aebafdece95 --- /dev/null +++ b/peps/pep-0751.rst @@ -0,0 +1,934 @@ +PEP: 751 +Title: A file format to list Python dependencies for installation reproducibility +Author: Brett Cannon +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 24-Jul-2024 +Replaces: 665 + +======== +Abstract +======== + +This PEP proposes a new file format for dependency specification +to enable reproducible installation in a Python environment. The format is +designed to be human-readable and machine-generated. Installers consuming the +file should be able to evaluate each package in question in isolation, with no +need for dependency resolution at install-time. + + +========== +Motivation +========== + +Currently, no standard exists to: + +- Specify what top-level dependencies should be installed into a Python + environment. +- Create an immutable record, such as a lock file, of which dependencies were + installed. + +Considering there are at least four well-known solutions to this problem in the +community (``pip freeze``, pip-tools_, Poetry_, and PDM_), there seems to be an +appetite for lock files in general. + +Those tools also vary in what locking scenarios they support. For instance, +``pip freeze`` and pip-tools only generate lock files for the current +environment while PDM and Poetry try to lock for *any* environment to some +degree. And none of them directly support locking to specific files to install +which can be important for some workflows. There's also concerns around the lack +of secure defaults in the face of supply chain attacks (e.g., always including +hashes for files). Finally, not all the formats are easy to audit to determine +what would be installed into an environment ahead of time. + +The lack of a standard also has some drawbacks. For instance, any tooling that +wants to work with lock files must choose which format to support, potentially +leaving users unsupported (e.g., if Dependabot_ chose not to support PDM, +support by cloud providers who can do dependency installations on your behalf, +etc.). + + +========= +Rationale +========= + +The format is designed so that a *locker* which produces the lock file +and an *installer* which consumes the lock file can be separate tools. This +allows for situations such as cloud hosting providers to use their own installer +that's optimized for their system which is independent of what locker the user +used to create their lock file. + +The file format is designed to be human-readable. This is +so that the contents of the file can be audited by a human to make sure no +undesired dependencies end up being included in the lock file. It is also to +facilitate easy understanding of what would be installed if the lock file +without necessitating running a tool, once again to help with auditing. Finally, +the format is designed so that viewing a diff of the file is easy by centralizing +relevant details. + +The file format is also designed to not require a resolver at install time. +Being able to analyze dependencies in isolation from one another when listed in +a lock file provides a few benefits. First, it supports auditing by making it +easy to figure out if a certain dependency would be installed for a certain +environment without needing to reference other parts of the file contextually. +It should also lead to faster installs which are much more frequent than +creating a lock file. Finally, the four tools mentioned in the Motivation_ +section either already implement this approach of evaluating dependencies in +isolation or have suggested they could (in +`Poetry's case `__). + + +----------------- +Locking Scenarios +----------------- + +The lock file format is designed to support two locking scenarios. The format +should also be flexible enough that adding support for other locking scenarios +is possible via a separate PEP. + + +Per-file Locking +================ + +*Per-file locking* operates under the premise that one wants to install exactly +the same files in any matching environment. As such, the lock file specifies the +with the files to install. There can be multiple environments specified in a +single file, each with their own set of files to install. By specifying the +exact files to install, installers avoid performing any resolution to decide what +to install. + +The motivation for this approach to locking is for those who have controlled +environments that they work with. For instance, if you have specific, controlled +development and production environments then you can use per-file locking to +make sure the **same** files are installed in both environments for everyone. +This is similar to what ``pip freeze`` and pip-tools_ +support, but with more strictness of the exact files as well as incorporating +support to specify the locked files for multiple environments in the same file. + + +Package Locking +=============== + +*Package locking* lists the packages and their versions that *may* apply to any +environment being installed for. The list of packages and their versions are +evaluated individually and independently from any other packages and versions +listed in the file. This allows installation to be linear -- read each package +and version and make an isolated decision as to whether it should be installed. +This avoids requiring the installer to perform a *resolution* (i.e. +determine what to install based on what else is to be installed). + +The motivation of this approach comes from +`PDM lock files `__. By listing the +potential packages and versions that may be installed, what's installed is +controlled in a way that's easy to reason about. This also allows for not +specifying the exact environments that would be supported by the lock file so +there's more flexibility for what environments are compatible with the lock +file. This approach supports scenarios like open-source projects that want to +lock what people should use to build the documentation without knowing upfront +what environments their contributors are working from. + +As already mentioned, this approach is supported by PDM_. Poetry_ has +`shown some interest `__. + + +============= +Specification +============= + +--------- +File Name +--------- + +A lock file MUST be named :file:`pylock.toml` or match the regular expression +``r"pylock\.(.+)\.toml"`` if a name for the lock file is desired or if multiple lock files exist. +The use of the ``.toml`` file extension is to make syntax highlighting in +editors easier and to reinforce the fact that the file format is meant to be +human-readable. The prefix and suffix of a named file MUST be lowercase for easy +detection and stripping off to find the name, e.g.:: + + if filename.startswith("pylock.") and filename.endswith(".toml"): + name = filename.removeprefix("pylock.").removesuffix(".toml") + +This PEP has no opinion as to the location of lock files (i.e. in the root or +the subdirectory of a project). + + +----------- +File Format +----------- + +The format of the file is TOML_. + +All keys listed below are required unless otherwise noted. If two keys are +mutually exclusive to one another, then one of the keys is required while the +other is disallowed. + + +``version`` +=========== + +- String +- The version of the lock file format. +- This PEP specifies the initial version -- and only valid value until future + updates to the standard change it -- as ``"1.0"``. + + +``hash-algorithm`` +================== + +- String +- The name of the hash algorithm used for calculating all hash values. +- Only a single hash algorithm is used for the entire file to allow the + ``[[package.files]]`` table to be written inline for readability and + compactness purposes by only listing a single hash value instead of multiple + values based on multiple hash algorithms. +- Specifying a single hash algorithm guarantees that an algorithm that the user + prefers is used consistently throughout the file without having to audit + each file hash value separately. +- Allows for updating the entire file to a new hash algorithm without running + the risk of accidentally leaving an old hash value in the file. +- :ref:`packaging:simple-repository-api-json` and the ``hashes`` dictionary of + of the ``files`` dictionary of the Project Details dictionary specifies what + values are valid and guidelines on what hash algorithms to use. +- Failure to validate any hash values for any file that is to be installed MUST + raise an error. + + +``dependencies`` +================ + +- Array of strings +- A listing of the `dependency specifiers`_ that act as the input to the lock file, + representing the direct, top-level dependencies to be installed. + + +``[[file-lock]]`` +================= + +- Array of tables +- Mutually exclusive with ``[package-lock]``. +- The array's existence implies the use of the per-file locking approach. +- An environment that meets all of the specified criteria in the table will be + considered compatible with the environment that was locked for. +- Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be + considered compatible for the same environment. +- In instances where there would be a conflict but the lock is still desired, + either separate lock files can be written or per-package locking can be used. +- Entries in array SHOULD be sorted by ``file-lock.name`` lexicographically. + + +``file-lock.name`` +------------------ + +- String +- A unique name within the array for the environment this table represents. + + +``[file-lock.marker-values]`` +----------------------------- + +- Optional +- Table of strings +- The keys represent the names of `environment markers`_ and the values are the + values for those markers. +- Compatibility is defined by the environment's values matching what is in the + table. +- Lockers SHOULD sort the keys lexicographically to minimize changes when + updating the file. + + +``file-lock.wheel-tags`` +------------------------ + +- Optional +- Array of strings +- An unordered array of `wheel tags`_ which must be supported by the environment. +- The array MAY not be exhaustive to allow for a smaller array as well as to + help prevent multiple ``[[file-lock]]`` tables being compatible with the + same environment by having one array being a strict subset of another + ``file-lock.wheel-tags`` entry in the same file's + ``[[file-lock]]`` tables. +- Lockers SHOULD sort the keys lexicographically to minimize changes when + updating the file. +- Lockers MUST NOT include + `compressed tag sets `__ + or duplicate tags for consistency across lockers and to simplify checking for + compatibility. + + +``[package-lock]`` +================== + +- Table +- Mutually exclusive with ``[[file-lock]]``. +- Signifies the use of the package locking approach. + + +``package-lock.requires-python`` +-------------------------------- + +- String +- Holds the `version specifiers`_ for Python version compatibility for the + overall package locking. +- Provides at-a-glance information to know if the lock file *may* apply to a + version of Python instead of having to scan the entire file to compile the + same information. + + +``[[package]]`` +=============== + +- Array of tables +- The array contains all data on the locked package versions. +- Lockers SHOULD record packages in order by ``package.name`` lexicographically + and ``package.version`` by the sort order for `version specifiers`_. +- Lockers SHOULD record keys in the same order as written in this PEP to + minimmize changes when updating. +- Designed so that relevant details as to why a package is included are + in one place to make diff reading easier. + + +``package.name`` +---------------- + +- String +- The `normalized name`_ of the package. +- Part of what's required to uniquely identify this entry. + + +``package.version`` +------------------- + +- String +- The version of the package. +- Part of what's required to uniquely identify this entry. + + +``package.multiple-entries`` +---------------------------- + +- Boolean +- If package locking via ``[package-lock]``, then the multiple entries for the + same package MUST be mutually exclusive via ``package.marker`` (this is not + required for per-file locking as the ``package.*.lock`` entries imply mutual + exclusivity). +- Aids in auditing by knowing that there are multiple entries for the same + package that may need to be considered. + + +``package.description`` +----------------------- + +- Optional +- String +- The package's ``Summary`` from its `core metadata`_. +- Useful to help understand why a package was included in the file based on its + purpose. + + +``package.simple-repo-package-url`` +----------------------------------- + +- Optional (although mutually exclusive with + ``package.files.simple-repo-package-url``) +- String +- Stores the `project detail`_ URL from the `Simple Repository API`_. +- Useful for generating Packaging URLs (aka PURLs). +- When possible, lockers SHOULD include this or + ``package.files.simple-repo-package-url`` to assist with generating + `software bill of materials`_ (aka SBOMs). + + +``package.marker`` +------------------ + +- Optional +- String +- The `environment markers`_ expression which specifies whether this package and + version applies to the environment. +- Only applicable via ``[package-lock]`` and the package locking scenario. +- The lack of this key means this package and version is required to be + installed. + + +``package.requires-python`` +--------------------------- + +- Optional +- String +- Holds the `version specifiers`_ for Python version compatibility for the + package and version. +- Useful for documenting why this package and version was included in the file. +- Also helps document why the version restriction in + ``package-lock.requires-python`` was chosen. +- It should not provide useful information for installers as it would be + captured by ``package-lock.requires-python`` and isn't relevant when + ``[[file-lock]]`` is used. + + +``package.dependents`` +---------------------- + +- Optional +- Array of strings +- A record of the packages that depend on this package and version. +- Useful for analyzing why a package happens to be listed in the file + for auditing purposes. +- This does not provide information which influences installers. + + +``package.dependencies`` +------------------------ + +- Optional +- Array of strings +- A record of the dependencies of the package and version. +- Useful in analyzing why a package happens to be listed in the file + for auditing purposes. +- This does not provide information which influences the installer as + ``[[file-lock]]`` specifies the exact files to use and ``[package-lock]`` + applicability is determined by ``package.marker``. + + +``package.direct`` +------------------ + +- Optional (defaults to ``false``) +- Boolean +- Represents whether the installation is via a `direct URL reference`_. + + +``[[package.files]]`` +--------------------- + +- Must be specified if ``[package.vcs]`` is not +- Array of tables +- Tables can be written inline. +- Represents the files to potentially install for the package and version. +- Entries in ``[[package.files]]`` SHOULD be lexicographically sorted by + ``package.files.name`` key to minimze changes in diffs. + + +``package.files.name`` +'''''''''''''''''''''' + +- String +- The file name. +- Necessary for installers to decide what to install when using package locking. + + +``package.files.lock`` +'''''''''''''''''''''' + +- Required when ``[[file-lock]]`` is used +- Array of strings +- An array of ``file-lock.name`` values which signify that the file is to be + installed when the corresponding ``[[file-lock]]`` table applies to the + environment. +- There MUST only be a single file with any one ``file-lock.name`` entry per + package, regardless of version. + + +``package.files.simple-repo-package-url`` +''''''''''''''''''''''''''''''''''''''''' + +- Optional (although mutually exclusive with + ``package.simple-repo-package-url``) +- String +- The value has the same meaning as ``package.simple-repo-package-url``. +- This key is available per-file to support :pep:`708` when some files override + what's provided by another `Simple Repository API`_ index. + + +``package.files.origin`` +'''''''''''''''''''''''' + +- Optional +- String +- URI where the file was found when the lock file was generated. +- Useful for documenting where the file came from and potentially where to look + for the file if not already downloaded/available. + + +``package.files.hash`` +'''''''''''''''''''''' + +- String +- The hash value of the file contents using the hash algorithm specified by + ``hash-algorithm``. +- Used by installers to verify the file contents match what the locker worked + with. + + +``[package.vcs]`` +----------------- + +- Must be specified if ``[[package.files]]`` is not (although may be specified + simultaneously with ``[[package.files]]``). +- Table representing the version control system containing the package and + version. + + +``package.vcs.type`` +'''''''''''''''''''' + +- String +- The type of version control system used. +- The valid values are specified by the + `registered VCSs `__ + of the direct URL data structure. + + +``package.vcs.origin`` +'''''''''''''''''''''' + +- String +- The URI of where the repository was located when the lock file was generated. + + +``package.vcs.commit`` +'''''''''''''''''''''' + +- String +- The commit ID for the repository which represents the package and version. +- The value MUST be immutable for the VCS for security purposes + (e.g. no Git tags). + + +``package.vcs.lock`` +'''''''''''''''''''' + +- Required when ``[[file-lock]]`` is used +- An array of strings +- An array of ``file-lock.name`` values which signify that the repository at the + specified commit is to be installed when the corresponding ``[[file-lock]]`` + table applies to the environment. +- A name in the array may only appear if no file listed in + ``package.files.lock`` contains the name for the same package, regardless of + version. + + +``package.directory`` +--------------------- + +- Optional and only valid when ``[package-lock]`` is specified +- String +- A local directory where a source tree for the package and version exists. +- Not valid under ``[[file-lock]]`` as this PEP does not make an attempt to + specify a mechanism for verifying file contents have not changed since locking + was performed. + + +``[[package.build-requires]]`` +------------------------------ + +- Optional +- An array of tables whose structure matches that of ``[[package]]``. +- Each entry represents a package and version to use when building the + enclosing package and version. +- The array is complete/locked like ``[[package]]`` itself (i.e. installers + follow the same installation procedure for ``[[package.build-requires]]`` as + ``[[package]]``) +- Selection of which entries to use for an environment as the same as + ``[[package]]`` itself, albeit only applying when installing the build + back-end and its dependencies. +- This helps with reproducibility of the building of a package by recording + either what was or would have been used if the locker needed to build the + package. +- If the installer and user choose to install from source and this array is + missing then the installer MAY choose to resolve what to install for building + at install time, otherwise the installer MUST raise an error. + + +``[package.tool]`` +------------------ + +- Optional +- Same usage as that of the equivalent table from the + `pyproject.toml specification`_. + + +``[tool]`` +========== + +- Optional +- Same usage as that of the equivalent table from the + `pyproject.toml specification`_. + + +------------------------ +Expectations for Lockers +------------------------ + +- When creating a lock file for ``[package-lock]``, the locker SHOULD read + the metadata of **all** files that end up being listed in + ``[[package.files]]`` to make sure all potential metadata cases are covered +- If a locker chooses not to check every file for its metadata, the tool MUST + either provide the user with the option to have all files checked (whether + that is opt-in or out is left up to the tool), or the user is somehow notified + that such a standards-violating shortcut is being taken (whether this is by + documentation or at runtime is left to the tool) +- Lockers MAY want to provide a way to let users provide the information + necessary to install for multiple environments at once when doing per-file + locking, e.g. supporting a JSON file format which specifies wheel tags and + marker values much like in ``[[file-lock]]`` for which multiple files can be + specified, which could then be directly recorded in the corresponding + ``[[file-lock]]`` table (if it allowed for unambiguous per-file locking + environment selection) + +.. code-block:: JSON + + { + "marker-values": {"": ""}, + "wheel-tags": [""] + } + + +--------------------------- +Expectations for Installers +--------------------------- + +- Installers MAY support installation of non-binary files + (i.e. source distributions, source trees, and VCS), but are not required to +- Installers MUST provide a way to avoid non-binary file installation for + reproducibility and security purposes +- Installers SHOULD make it opt-in to use non-binary file installation to + facilitate a secure-by-default approach +- Under per-file locking, if what to install is ambiguous then the installer + MUST raise an error + + +Installing for per-file locking +=============================== + +An example workflow is: + +- Iterate through each ``[[file-lock]]`` table to find the one that applies to + the environment being installed for +- If no compatible environment is found an error MUST be raised +- If multiple environments are found to be compatible then an error MUST be raised +- For the compatible environment, iterate through each entry in ``[[package]]`` +- For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look + for any files with ``file-lock.name`` listed in ``package.files.lock`` +- If a file is found with a matching lock name, add it to the list of candidate + files to install and move on to the next ``[[package]]`` entry +- If no file is found then check if ``package.vcs.lock`` contains a match (no + match is also acceptable) +- If a ``[[package.files]]`` contains multiple matching entries an error MUST + be raised due to ambiguity for what is to be installed +- If multiple ``[[package]]`` entries for the same package have matching files + an error MUST be raised due to ambiguity for what is to be installed +- Find and verify the candidate files and/or VCS entries based on their hash or + commit ID as appropriate +- If a source distribution or VCS was selected and + ``[[package.build-requires]]`` exists, then repeat the above process as + appropriate to install the build dependencies necessary to build the package +- Install the candidate files + + +Installing for package locking +============================== + +An example workflow is: + +- Verify that the environment is compatible with + ``package-lock.requires-python``; if it isn't an error MUST be raised +- Iterate through each entry in ``[package]]`` +- For each entry, if there's a ``package.marker`` key, evaluate the expression + + - If the expression is false, then move on + - Otherwise the package entry must be installed somehow +- Iterate through the files listed in ``[[package.files]]``, looking for the + "best" file to install +- If no file is found, check for ``[package.vcs]`` +- If no match is found, an error MUST be raised +- Find and verify the selected files and/or VCS entries based on their hash or + commit ID as appropriate +- If the match is a source distribution or VCS and + ``[[package.build-requires]]`` is provided, repeat the above as appropriate to + build the package +- Install the selected files + + +======================= +Backwards Compatibility +======================= + +Because there is no preexisting lock file format, there are no explicit +backwards-compatibility concerns in terms of Python packaging standards. + +As for packaging tools themselves, that will be a per-tool decision. For tools +that don't document their lock file format, they could choose to simply start +using the format internally and then transition to saving their lock files with +a name supported by this PEP. For tools with a preexisting, documented format, +they could provide an option to choose which format to emit. + + +===================== +Security Implications +===================== + +The hope is that by standardizing on a lock file format that starts from a +security-first posture it will help make overall packaging installation safer. +However, this PEP does not solve all potential security concerns. + +One potential concern is tampering with a lock file. If a lock file is not kept +in source control and properly audited, a bad actor could change the file in +nefarious ways (e.g. point to a malware version of a package). Tampering could +also occur in transit to e.g. a cloud provider who will perform an installation +on the user's behalf. Both could be mitigated by signing the lock file either +within the file in a ``[tool]`` entry or via a side channel external to the lock +file itself. + +This PEP does not do anything to prevent a user from installing an incorrect +package. While including many details to help in auditing a package's inclusion, +there isn't any mechanism to stop e.g. name confusion attacks via typosquatting. +Lockers may be able to provide some UX to help with this (e.g. by providing +download counts for a package). + + +================= +How to Teach This +================= + +Users should be informed that when they ask to install some package, that +package may have its own dependencies, those dependencies may have dependencies, +and so on. Without writing down what gets installed as part of installing the +package they requested, things could change from underneath them (e.g. package +versions). Changes to the underlying dependencies can lead to accidental +breakage of their code. Lock files help deal with that by providing a way to +write down what was installed. + +Having what to install written down also helps in collaborating with others. By +agreeing to a lock file's contents, everyone ends up with the same packages +installed. This helps make sure no one relies on e.g. an API that's only +available in a certain version that not everyone working on the project has +installed. + +Lock files also help with security by making sure you always get the same files +installed and not a malicious one that someone may have slipped in. It also +lets one be more deliberate in upgrading their dependencies and thus making sure +the change is on purpose and not one slipped in by a bad actor. + + +======================== +Reference Implementation +======================== + +A rough proof-of-concept for per-file locking can be found at +https://github.com/brettcannon/mousebender/tree/pep. An example lock file can +be seen at +https://github.com/brettcannon/mousebender/blob/pep/pylock.example.toml. + +For per-package locking, PDM_ indirectly proves the approach works as this PEP +maintains equivalent data as PDM does for its lock files (whose format was +inspired by Poetry_). Some of the details of PDM's approach are covered in +https://frostming.com/en/2024/pdm-lockfile/ and +https://frostming.com/en/2024/pdm-lock-strategy/. + + +============== +Rejected Ideas +============== + +---------------------------- +Only support package locking +---------------------------- + +At one point it was suggested to skip per-file locking and only support package +locking as the former was not explicitly supported in the larger Python +ecosystem while the latter was. But because this PEP has taken the position +that security is important and per-file locking is the more secure of the two +options, leaving out per-file locking was never considered. + + +------------------------------------------------------------------------------------- +Specifying a new core metadata version that requires consistent metadata across files +------------------------------------------------------------------------------------- + +At one point, to handle the issue of metadata varying between files and thus +require examining every released file for a package and version for accurate +locking results, the idea was floated to introduce a new core metadata version +which would require all metadata for all wheel files be the same for a single +version of a package. Ultimately, though, it was deemed unnecessary as this PEP +will put pressure on people to make files consistent for performance reasons or +to make indexes provide all the metadata separate from the wheel files +themselves. As well, there's no easy enforcement mechanism, and so community +expectation would work as well as a new metadata version. + + +------------------------------------------- +Have the installer do dependency resolution +------------------------------------------- + +In order to support a format more akin to how Poetry worked when this PEP was +drafted, it was suggested that lockers effectively record the packages and their +versions which may be necessary to make an install work in any possible +scenario, and then the installer resolves what to install. But that complicates +auditing a lock file by requiring much more mental effort to know what packages +may be installed in any given scenario. Also, one of the Poetry developers +`suggested `__ +that markers as represented in the package locking approach of this PEP may be +sufficient to cover the needs of Poetry. Not having the installer do a +resolution also simplifies their implementation, centralizing complexity in +lockers. + + +----------------------------------------- +Requiring specific hash algorithm support +----------------------------------------- + +It was proposed to require a baseline hash algorithm for the files. This was +rejected as no other Python packaging specification requires specific hash +algorithm support. As well, the minimum hash algorithm suggested may eventually +become an outdated/unsafe suggestion, requiring further updates. In order to +promote using the best algorithm at all times, no baseline is provided to avoid +simply defaulting to the baseline in tools without considering the security +ramifications of that hash algorithm. + + +----------- +File naming +----------- + +Using ``*.pylock.toml`` as the file name +======================================== + +It was proposed to put the ``pylock`` constant part of the file name after the +identifier for the purpose of the lock file. It was decided not to do this so +that lock files would sort together when looking at directory contents instead +of purely based on their purpose which could spread them out in a directory. + + +Using ``*.pylock`` as the file name +=================================== + +Not using ``.toml`` as the file extension and instead making it ``.pylock`` +itself was proposed. This was decided against so that code editors would know +how to provide syntax highlighting to a lock file without having special +knowledge about the file extension. + + +Not having a naming convention for the file +=========================================== + +Having no requirements or guidance for a lock file's name was considered, but +ultimately rejected. By having a standardized naming convention it makes it easy +to identify a lock file for both a human and a code editor. This helps +facilitate discovery when e.g. a tool wants to know all of the lock files that +are available. + + +----------- +File format +----------- + +Use JSON over TOML +================== + +Since having a format that is machine-writable was a goal of this PEP, it was +suggested to use JSON. But it was deemed less human-readable than TOML while +not improving on the machine-writable aspect enough to warrant the change. + + +Use YAML over TOML +================== + +Some argued that YAML met the machine-writable/human-readable requirement in a +better way than TOML. But as that's subjective and ``pyproject.toml`` already +existed as the human-writable file used by Python packaging standards it was +deemed more important to keep using TOML. + + +---------- +Other keys +---------- + +Multiple hashes per file +======================== + +An initial version of this PEP proposed supporting multiple hashes per file. The +idea was to allow one to choose which hashing algorithm they wanted to go with +when installing. But upon reflection it seemed like an unnecessary complication +as there was no guarantee the hashes provided would satisfy the user's needs. +As well, if the single hash algorithm used in the lock file wasn't sufficient, +rehashing the files involved as a way to migrate to a different algorithm didn't +seem insurmountable. + + +Hashing the contents of the lock file itself +============================================ + +Hashing the contents of the bytes of the file and storing hash value within the +file itself was proposed at some point. This was removed to make it easier +when merging changes to the lock file as each merge would have to recalculate +the hash value to avoid a merge conflict. + +Hashing the semantic contents of the file was also proposed, but it would lead +to the same merge conflict issue. + +Regardless of which contents were hashed, either approach could have the hash +value stored outside of the file if such a hash was desired. + + +Recording the creation date of the lock file +============================================ + +To know how potentially stale the lock file was, an earlier proposal suggested +recording the creation date of the lock file. But for some same merge conflict +reasons as storing the hash of the file contents, this idea was dropped. + + +Recording the package indexes used +================================== + +Recording what package indexes were used by the locker to decide what to lock +for was considered. In the end, though, it was rejected as it was deemed +unnecessary bookkeeping. + + +=========== +Open Issues +=========== + +N/A + + +================ +Acknowledgements +================ + +Thanks to everyone who participated in the discussions in +https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/, +especially Alyssa Coghlan who probably caused the biggest structural shifts from +the initial proposal. + +Also thanks to Randy Döring, Seth Michael Larson, Paul Moore, and Ofek Lev for +providing feedback on a draft version of this PEP. + + +========= +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + + +.. _core metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ +.. _Dependabot: https://docs.github.com/en/code-security/dependabot +.. _dependency specifiers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ +.. _direct URL reference: https://packaging.python.org/en/latest/specifications/direct-url/ +.. _environment markers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers +.. _normalized name: https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization +.. _PDM: https://pypi.org/project/pdm/ +.. _pip-tools: https://pypi.org/project/pip-tools/ +.. _Poetry: https://python-poetry.org/ +.. _project detail: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail +.. _pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification +.. _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ +.. _software bill of materials: https://www.cisa.gov/sbom +.. _TOML: https://toml.io/ +.. _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ +.. _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ From 5b4e2ab702743fec5d7a6bcc80e7ae67f1e6fd2b Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Thu, 25 Jul 2024 14:32:18 +0200 Subject: [PATCH 19/27] PEP 743: Rewrite to hide (soft-)deprecated API (GH-3869) Co-authored-by: Victor Stinner --- .github/CODEOWNERS | 2 +- peps/pep-0743.rst | 436 ++++++++++++++++++++++++++++++++------------- 2 files changed, 313 insertions(+), 125 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 45741d03c2c..7ba65e30704 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -621,7 +621,7 @@ peps/pep-0738.rst @encukou peps/pep-0740.rst @dstufft peps/pep-0741.rst @vstinner peps/pep-0742.rst @JelleZijlstra -peps/pep-0743.rst @vstinner +peps/pep-0743.rst @vstinner @encukou peps/pep-0744.rst @brandtbucher peps/pep-0745.rst @hugovk peps/pep-0746.rst @JelleZijlstra diff --git a/peps/pep-0743.rst b/peps/pep-0743.rst index 39d9039eb1a..3e202c88b48 100644 --- a/peps/pep-0743.rst +++ b/peps/pep-0743.rst @@ -1,10 +1,11 @@ PEP: 743 Title: Add Py_COMPAT_API_VERSION to the Python C API -Author: Victor Stinner +Author: Victor Stinner , + Petr Viktorin , Status: Draft Type: Standards Track Created: 11-Mar-2024 -Python-Version: 3.13 +Python-Version: 3.14 .. highlight:: c @@ -12,177 +13,361 @@ Python-Version: 3.13 Abstract ======== -Add ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros -to opt-in for planned incompatible C API changes in a C extension. -Maintainers can decide when they make their C extension compatible -and also decide which future Python version they want to be compatible -with. +Add ``Py_COMPAT_API_VERSION`` C macro that hides some deprecated and +soft-deprecated symbols, allowing users to opt out of using API with known +issues that other API solves. +The macro is versioned, allowing users to update (or not) on their own pace. +Also, add namespaced alternatives for API without the ``Py_`` prefix, +and soft-deprecate the original names. -Rationale -========= -Python releases enforce C API changes -------------------------------------- +Motivation +========== -Every Python 3.x release has a long list of C API changes, including -incompatible changes. C extensions have to be updated to work on the -newly released Python. +Some of Python's C API has flaws that are only obvious in hindsight. -Some incompatible changes are driven by new features: they cannot be -avoided, unless we decide to not add these features. Other reasons: +If an API prevents adding features or optimizations, or presents a serious +security risk or maintenance burden, we can deprecate and remove it as +described in :pep:`387`. -* Remove deprecated API (see :pep:`387`). -* Ease the implementation of another change. -* Change or remove error-prone API. +However, this leaves us with some API that has “sharp edges” -- it works fine +for its current users, but should be avoided in new code. +For example: -Currently, there is no middle ground between "not change the C API" and -"incompatible C API changes impact everybody". Either a C extension is -updated or the new Python version cannot be used. Such all-or-nothing -deal does not satisfy C extension maintainers nor C extensions users. +- API that cannot signal an exception, so failures are either ignored or + exit the process with a fatal error. For example ``PyObject_HasAttr``. +- API that is not thread-safe, for example by borrowing references from + mutable objects, or exposing unfinished mutable objects. For example + ``PyDict_GetItemWithError``. +- API with names that don't use the ``Py``/``_Py`` prefix, and so can clash + with other code. For example: ``setter``. +It is important to note that despite such flaws, it's usually possible +to use the API correctly. For example, in a single-threaded environment, +thread safety is not an issue. +We do not want to break working code, even if it uses API that would be wrong +in some -- or even *most* -- other contexts. -Limited C API -------------- - -The limited C API is versioned: the ``Py_LIMITED_API`` macro can be set -to a Python version to select which API is available. On the Python -side, it allows introducing incompatible changes at a specific -``Py_LIMITED_API`` version. For example, if ``Py_LIMITED_API`` is set to -Python 3.11 or newer, the ```` is no longer included by -``Python.h``, whereas C extensions targeting Python 3.10 are not -affected. +On the other hand, we want to steer users away from such “undesirable” API +in *new* code, especially if a safer alternative exists. -The difference here is that upgrading Python does not change if -```` is included or not, but updating ``Py_LIMITED_API`` does. -Updating ``Py_LIMITED_API`` is an deliberate action made by the C -extension maintainer. It gives more freedom to decide **when** the -maintainer is ready to deal with the latest batch of incompatible -changes. -A similar version can be used with the regular (non-limited) C API. +Adding the ``Py`` prefix +------------------------ +Some names defined in CPython headers is not namespaced: it that lacks the +``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations). +For example, we declare a function type named simply ``setter``. -Deprecation and compiler warnings ---------------------------------- +While such names are not exported in the ABI (as checked by ``make smelly``), +they can clash with user code and, more importantly, with libraries linked +to third-party extensions. -Deprecated functions are marked with ``Py_DEPRECATED()``. Using a -deprecated function emits a compiler warning. +While it would be possible to provide namespaced aliases and (soft-)deprecate +these names, the only way to make them not clash with third-party code is to +not define them in Python headers at all. -The problem is that ``pip`` and ``build`` tools hide compiler logs by -default, unless a build fails. Moreover, it's easy to miss a single -warning in the middle of hundred lines of logs. -Schedule changes ----------------- +Rationale +========= -Currently, there is no way to schedule a C API change: announce it but -also provide a way to maintainers to test their C extensions with the -change. Either a change is not made, or everybody must update their code -if they want to update Python. +We want to allow an easy way for users to avoid “undesirable” API if they +choose to do so. + +It might be be sufficient to leave this to third-party linters. +For that we'd need a good way to expose a list of (soft-)deprecated +API to such linters. +While adding that, we can -- rather easily -- do the linter's job directly +in CPython headers, avoiding the neel for an extra tool. +Unlike Python, C makes it rather easy to limit available API -- for a whole +project or for each individual source file -- by having users define +an “opt-in” macro. + +We already do something similar with ``Py_LIMITED_API``, which limits the +available API to a subset that compiles to stable ABI. (In hindsight, we should +have used a different macro name for that particular kind of limiting, but it's +too late to change that now.) + +To prevent working code from breaking as we identify more “undesirable” API +and add safer alternatives to it, the opt-in macro should be *versioned*. +Users can choose a version they need based on their compatibility requirements, +and update it at their own pace. + +To be clear, this mechanism is *not* a replacement for deprecation. +Deprecation is for API that prevents new features or optimizations, or +presents a security risk or maintenance burden. +This mechanism, on the other hand, is meant for cases where “we found +a slightly better way of doing things” -- perhaps one that's harder to misuse, +or just has a less misleading name. +(On a lighter note: many people configure a code quality checker to shout at +them about the number of blank lines between functions. Let's help them +identify more substantial “code smells”!) + +The proposed macro does not *change* any API definitions; it only *hides* them. +So, if code compiles with the macro, it'll also compile without it, with +identical behaviour. +This has implications for core devs: to deal with undesirable behaviour, +we'll need to introduce new, better API, and *then* discourage the old one. +In turn, this implies that we should look at an individual API and fix all its +known issues at once, rather than do codebase-wide sweeps for a single kind of +issue, so that we avoid multiple renames of the same function. + + +Adding the ``Py`` prefix +------------------------ + +An opt-in macro allows us to omit definitions that could clash with +third-party libraries. Specification ============= -New macros ----------- - -Add new ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` -macros. They can be set to test if a C extension is prepared for future -C API changes: compatible with future Python versions. +We introduce a ``Py_COMPAT_API_VERSION`` macro. +If this macro is defined before ``#include ``, some API definitions +-- as described below -- will be omitted from the Python header files. -The ``Py_COMPAT_API_VERSION`` macro can be set to a specific Python -version. For example, ``Py_COMPAT_API_VERSION=0x030e0000`` tests C API -changes scheduled in Python 3.14. +The macro only omits complete top-level definitions exposed from ````. +Other things (the ABI, structure definitions, macro expansions, static inline +function bodies, etc.) are not affected. -If the ``Py_COMPAT_API_VERSION`` macro is set to -``Py_COMPAT_API_VERSION_MAX``, all scheduled C API changes are tested at -once. +The C API working group (:pep:`731`) has authority over the set of omitted +definitions. -If the ``Py_COMPAT_API_VERSION`` macro is not set, it is to -``PY_VERSION_HEX`` by default. +The set of omitted definitions will be tied to a particular feature release +of CPython, and is finalized in each 3.x.0 Beta 1 release. +In rare cases, entries can be removed (i.e. made available for use) at any +time. -The ``Py_COMPAT_API_VERSION`` macro can be set in a single C file or for -a whole project in compiler flags. The macro does not affected other -projects or Python itself. +The macro should be defined to a version in the format used by +``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields +set to zero. +For example, to omit API deemed undesirable in 3.14.0b1, users should define +``Py_COMPAT_API_VERSION`` to ``0x030e0000``. -Example in Python ------------------ +Requirements for omitted API +---------------------------- -For example, the ``PyImport_ImportModuleNoBlock()`` function is -deprecated in Python 3.13 and scheduled for removal in Python 3.15. The -function can be declared in the Python C API with the following -declaration: +An API that is omitted with ``Py_COMPAT_API_VERSION`` must: -.. code-block:: c +- be soft-deprecated (see :pep:`387`); +- for all known use cases of the API, have a documented alternative + or workaround; +- have tests to ensure it keeps working (except for 1:1 renames using + ``#define`` or ``typedef``); +- be documented (except if it was never mentioned in previous versions of the + documentation); and +- be approved by the C API working group. (The WG may give blanket approvals + for groups of related API; see *Initial set* below for examples.) - #if Py_COMPAT_API_VERSION < 0x030f0000 - Py_DEPRECATED(3.13) PyAPI_FUNC(PyObject *) PyImport_ImportModuleNoBlock( - const char *name /* UTF-8 encoded string */ - ); - #endif +Note that ``Py_COMPAT_API_VERSION`` is meant for API that can be trivially +replaced by a better alternative. +API without a replacement should generally be deprecated instead. -If ``if Py_COMPAT_API_VERSION`` is equal to or greater than Python 3.15 -(``0x030f0000``), the ``PyImport_ImportModuleNoBlock()`` function is not -declared, and so using it fails with a build error. -Goals ------ +Location +-------- -* Reduce the number of C API changes affecting C extensions when - updating Python. -* When testing C extensions (for example, optional CI test), - ``Py_COMPAT_API_VERSION`` can be set to ``Py_COMPAT_API_VERSION_MAX`` - to detect future incompatibilities. For mandatory tests, it is - recommended to set ``Py_COMPAT_API_VERSION`` to a specific Python - version. -* For core developers, make sure that the C API can still evolve - without being afraid of breaking an unknown number of C extensions. +All API definitions omitted by ``Py_COMPAT_API_VERSION`` will be moved to +a new header, ``Include/legacy.h``. -Non-goals ---------- +This is meant to help linter authors compile lists, so they can flag the API +with warnings rather than errors. -* Freeze the API forever: this is not the stable ABI. For example, - deprecated functions will continue to be removed on a regular basis. -* C extensions maintainers not using ``Py_COMPAT_API_VERSION`` will - still be affected by C API changes when updating Python. -* Provide a stable ABI: the macro only impacts the regular (non-limited) - API. -* Silver bullet solving all C API issues. +Note that for simple renaming of source-only constructs (macros, types), we +expect names to be omitted in the same version -- or the same PR -- that adds +a replacement. +This means that the original definition will be renamed, and a ``typedef`` +or ``#define`` for the old name added to ``Include/legacy.h``. -Examples of ``Py_COMPAT_API_VERSION`` usages -============================================ +Documentation +------------- -* Remove deprecated functions. -* Remove deprecated structure members, such as - ``PyBytesObject.ob_shash``. -* Remove a standard ``#include``, such as ``#include ``, - from ````. -* Change the behavior of a function or a macro. For example, calling - ``PyObject_SetAttr(obj, name, NULL)`` can fail, to enforce the usage - of the ``PyObject_DelAttr()`` function instead to delete an attribute. +Documentation for omitted API should generally: + +- appear after the recommended replacement, +- reference the replacement (e.g. “Similar to X, but
”), and +- focus on differences from the replacement and migration advice. + +Exceptions are possible if there is a good reason for them. + + +Initial set +----------- + +The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to +``0x030e0000`` (3.14) or greater: + +- Omit API returning borrowed references: + + ==================================== ============================== + Omitted API Replacement + ==================================== ============================== + ``PyDict_GetItem()`` ``PyDict_GetItemRef()`` + ``PyDict_GetItemString()`` ``PyDict_GetItemStringRef()`` + ``PyImport_AddModule()`` ``PyImport_AddModuleRef()`` + ``PyList_GetItem()`` ``PyList_GetItemRef()`` + ==================================== ============================== + +- Omit deprecated APIs: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``PY_FORMAT_SIZE_T`` ``"z"`` + ``PY_UNICODE_TYPE`` ``wchar_t`` + ``PyCode_GetFirstFree()`` ``PyUnstable_Code_GetFirstFree()`` + ``PyCode_New()`` ``PyUnstable_Code_New()`` + ``PyCode_NewWithPosOnlyArgs()`` ``PyUnstable_Code_NewWithPosOnlyArgs()`` + ``PyImport_ImportModuleNoBlock()`` ``PyImport_ImportModule()`` + ``PyMem_DEL()`` ``PyMem_Free()`` + ``PyMem_Del()`` ``PyMem_Free()`` + ``PyMem_FREE()`` ``PyMem_Free()`` + ``PyMem_MALLOC()`` ``PyMem_Malloc()`` + ``PyMem_NEW()`` ``PyMem_New()`` + ``PyMem_REALLOC()`` ``PyMem_Realloc()`` + ``PyMem_RESIZE()`` ``PyMem_Resize()`` + ``PyModule_GetFilename()`` ``PyModule_GetFilenameObject()`` + ``PyOS_AfterFork()`` ``PyOS_AfterFork_Child()`` + ``PyObject_DEL()`` ``PyObject_Free()`` + ``PyObject_Del()`` ``PyObject_Free()`` + ``PyObject_FREE()`` ``PyObject_Free()`` + ``PyObject_MALLOC()`` ``PyObject_Malloc()`` + ``PyObject_REALLOC()`` ``PyObject_Realloc()`` + ``PySlice_GetIndicesEx()`` (two calls; see current docs) + ``PyThread_ReInitTLS()`` (no longer needed) + ``PyThread_create_key()`` ``PyThread_tss_alloc()`` + ``PyThread_delete_key()`` ``PyThread_tss_free()`` + ``PyThread_delete_key_value()`` ``PyThread_tss_delete()`` + ``PyThread_get_key_value()`` ``PyThread_tss_get()`` + ``PyThread_set_key_value()`` ``PyThread_tss_set()`` + ``PyUnicode_AsDecodedObject()`` ``PyUnicode_Decode()`` + ``PyUnicode_AsDecodedUnicode()`` ``PyUnicode_Decode()`` + ``PyUnicode_AsEncodedObject()`` ``PyUnicode_AsEncodedString()`` + ``PyUnicode_AsEncodedUnicode()`` ``PyUnicode_AsEncodedString()`` + ``PyUnicode_IS_READY()`` (no longer needed) + ``PyUnicode_READY()`` (no longer needed) + ``PyWeakref_GET_OBJECT()`` ``PyWeakref_GetRef()`` + ``PyWeakref_GetObject()`` ``PyWeakref_GetRef()`` + ``Py_UNICODE`` ``wchar_t`` + ``_PyCode_GetExtra()`` ``PyUnstable_Code_GetExtra()`` + ``_PyCode_SetExtra()`` ``PyUnstable_Code_SetExtra()`` + ``_PyDict_GetItemStringWithError()`` ``PyDict_GetItemStringRef()`` + ``_PyEval_RequestCodeExtraIndex()`` ``PyUnstable_Eval_RequestCodeExtraIndex()`` + ``_PyHASH_BITS`` ``PyHASH_BITS`` + ``_PyHASH_IMAG`` ``PyHASH_IMAG`` + ``_PyHASH_INF`` ``PyHASH_INF`` + ``_PyHASH_MODULUS`` ``PyHASH_MODULUS`` + ``_PyHASH_MULTIPLIER`` ``PyHASH_MULTIPLIER`` + ``_PyObject_EXTRA_INIT`` (no longer needed) + ``_PyThreadState_UncheckedGet()`` ``PyThreadState_GetUnchecked()`` + ``_PyUnicode_AsString()`` ``PyUnicode_AsUTF8()`` + ``_Py_HashPointer()`` ``Py_HashPointer()`` + ``_Py_T_OBJECT`` ``Py_T_OBJECT_EX`` + ``_Py_WRITE_RESTRICTED`` (no longer needed) + ==================================== ============================== + +- Soft-deprecate and omit APIs: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``PyDict_GetItemWithError()`` ``PyDict_GetItemRef()`` + ``PyDict_SetDefault()`` ``PyDict_SetDefaultRef()`` + ``PyMapping_HasKey()`` ``PyMapping_HasKeyWithError()`` + ``PyMapping_HasKeyString()`` ``PyMapping_HasKeyStringWithError()`` + ``PyObject_HasAttr()`` ``PyObject_HasAttrWithError()`` + ``PyObject_HasAttrString()`` ``PyObject_HasAttrStringWithError()`` + ==================================== ============================== + +- Omit ```` legacy API: + + The header file ``structmember.h``, which is not included from ```` + and must be included separately, will ``#error`` if + ``Py_COMPAT_API_VERSION`` is defined. + This affects the following API: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``T_SHORT`` ``Py_T_SHORT`` + ``T_INT`` ``Py_T_INT`` + ``T_LONG`` ``Py_T_LONG`` + ``T_FLOAT`` ``Py_T_FLOAT`` + ``T_DOUBLE`` ``Py_T_DOUBLE`` + ``T_STRING`` ``Py_T_STRING`` + ``T_OBJECT`` (``tp_getset``; docs to be written) + ``T_CHAR`` ``Py_T_CHAR`` + ``T_BYTE`` ``Py_T_BYTE`` + ``T_UBYTE`` ``Py_T_UBYTE`` + ``T_USHORT`` ``Py_T_USHORT`` + ``T_UINT`` ``Py_T_UINT`` + ``T_ULONG`` ``Py_T_ULONG`` + ``T_STRING_INPLACE`` ``Py_T_STRING_INPLACE`` + ``T_BOOL`` ``Py_T_BOOL`` + ``T_OBJECT_EX`` ``Py_T_OBJECT_EX`` + ``T_LONGLONG`` ``Py_T_LONGLONG`` + ``T_ULONGLONG`` ``Py_T_ULONGLONG`` + ``T_PYSSIZET`` ``Py_T_PYSSIZET`` + ``T_NONE`` (``tp_getset``; docs to be written) + ``READONLY`` ``Py_READONLY`` + ``PY_AUDIT_READ`` ``Py_AUDIT_READ`` + ``READ_RESTRICTED`` ``Py_AUDIT_READ`` + ``PY_WRITE_RESTRICTED`` (no longer needed) + ``RESTRICTED`` ``Py_AUDIT_READ`` + ==================================== ============================== + +- Omit soft deprecated macros: + + ====================== ===================================== + Omitted Macros Replacement + ====================== ===================================== + ``Py_IS_NAN()`` ``isnan()`` (C99+ ````) + ``Py_IS_INFINITY()`` ``isinf(X)`` (C99+ ````) + ``Py_IS_FINITE()`` ``isfinite(X)`` (C99+ ````) + ``Py_MEMCPY()`` ``memcpy()`` (C ````) + ====================== ===================================== + +- Soft-deprecate and omit typedefs without the ``Py``/``_Py`` prefix + (``getter``, ``setter``, ``allocfunc``, 
), in favour of *new* ones + that add the prefix (``Py_getter`` , etc.) + +- Soft-deprecate and omit macros without the ``Py``/``_Py`` prefix + (``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, 
), + favour of *new* ones that add the prefix (``Py_METH_O`` , etc.). + +- Any others approved by the C API workgroup + + +If any of these proposed replacements, or associated documentation, +are not added in time for 3.14.0b1, they'll be omitted with later versions +of ``Py_COMPAT_API_VERSION``. +(We expect this for macros generated by ``configure``: ``HAVE_*``, ``WITH_*``, +``ALIGNOF_*``, ``SIZEOF_*``, and several without a common prefix.) Implementation ============== -* `Issue gh-116587 `_ -* PR: `Add Py_COMPAT_API_VERSION and Py_COMPAT_API_VERSION_MAX macros - `_ +TBD + + +Open issues +=========== + +The name ``Py_COMPAT_API_VERSION`` was taken from the earlier PEP; +it doesn't fit this version. Backwards Compatibility ======================= -There is no impact on backward compatibility. - -Adding ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` -macros has no effect on backward compatibility. Only developers setting -the ``Py_COMPAT_API_VERSION`` macro in their project will be impacted by -effects of this macro which is the expected behavior. +The macro is backwards compatible. +Developers can introduce and update the macro on their own pace, potentially +for one source file at a time. Discussions @@ -195,6 +380,9 @@ Discussions with no known issues `_ (June 2023) +* `Finishing the Great Renaming + `_ + (May 2024) Prior Art From 5ef5ca00ebca0270b4528bd12c4bbea60118e58e Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 25 Jul 2024 11:59:04 -0700 Subject: [PATCH 20/27] PEP 751: Add Discussions-To and Post-History (#3872) --- peps/pep-0751.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index aebafdece95..5584d4bf268 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -1,10 +1,12 @@ PEP: 751 Title: A file format to list Python dependencies for installation reproducibility Author: Brett Cannon +Discussions-To: https://discuss.python.org/t/59173 Status: Draft Type: Standards Track Topic: Packaging Created: 24-Jul-2024 +Post-History: `25-Jul-2024 `__ Replaces: 665 ======== @@ -92,8 +94,8 @@ Per-file Locking ================ *Per-file locking* operates under the premise that one wants to install exactly -the same files in any matching environment. As such, the lock file specifies the -with the files to install. There can be multiple environments specified in a +the same files in any matching environment. As such, the lock file specifies +what files to install. There can be multiple environments specified in a single file, each with their own set of files to install. By specifying the exact files to install, installers avoid performing any resolution to decide what to install. From b7bf1ca2a461a5a601974622481ea09d4eaf8acd Mon Sep 17 00:00:00 2001 From: Karolina Surma <33810531+befeleme@users.noreply.github.com> Date: Thu, 25 Jul 2024 23:07:53 +0200 Subject: [PATCH 21/27] PEP 639: Incorporate the latest discussion feedback (#3866) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Remove the requirement of license-files defaults * Cover all rejected subkeysideas in one paragraph * Change the deprecation policy around classifiers * Flatten the value of the license-files key, only globs are specified * Update the Rejected ideas to match the current license-files proposal --------- Co-authored-by: Miro Hrončok --- peps/pep-0639.rst | 108 +++------ peps/pep-0639/appendix-examples.rst | 24 +- peps/pep-0639/appendix-rejected-ideas.rst | 279 ++++------------------ 3 files changed, 94 insertions(+), 317 deletions(-) diff --git a/peps/pep-0639.rst b/peps/pep-0639.rst index eb9e941aa26..433045b6437 100644 --- a/peps/pep-0639.rst +++ b/peps/pep-0639.rst @@ -451,22 +451,18 @@ in the ``Classifier`` :term:`Core Metadata field` (`described in the Core Metadata specification `__) is deprecated and replaced by the more precise ``License-Expression`` field. -If the ``License-Expression`` field is present, build tools SHOULD and -publishing tools MUST raise an error if one or more license classifiers +If the ``License-Expression`` field is present, build tools MAY raise an error +if one or more license classifiers is included in a ``Classifier`` field, and MUST NOT add such classifiers themselves. -Otherwise, if this field contains a license classifier, build tools MAY -and publishing tools SHOULD issue a warning informing users such classifiers +Otherwise, if this field contains a license classifier, +tools MAY issue a warning informing users such classifiers are deprecated, and recommending ``License-Expression`` instead. For compatibility with existing publishing and installation processes, the presence of license classifiers SHOULD NOT raise an error unless ``License-Expression`` is also provided. -For all newly-uploaded distributions that include a -``License-Expression`` field, the `Python Package Index (PyPI) `__ MUST -reject any that also specify any license classifiers. - New license classifiers MUST NOT be `added to PyPI `__; users needing them SHOULD use the ``License-Expression`` field instead. License classifiers may be removed from a new version of the specification @@ -524,104 +520,62 @@ paths in the project source tree relative to ``pyproject.toml`` to file(s) containing licenses and other legal notices to be distributed with the package. It corresponds to the ``License-File`` fields in the Core Metadata. -Its value is a table, which if present MUST contain one of two optional, -mutually exclusive subkeys, ``paths`` and ``globs``; if both are specified, -tools MUST raise an error. Both are arrays of strings; the ``paths`` subkey -contains verbatim file paths, and the ``globs`` subkey valid glob patterns, -which MUST be parsable by the ``glob`` `module `__ in the -Python standard library. - +Its value is an array of strings which MUST contain valid glob patterns, +as specified below. +The glob patterns MAY contain special glob characters: ``*``, ``?``, ``**`` +and character ranges: ``[]``, and tools MUST support them. Path delimiters MUST be the forward slash character (``/``), and parent directory indicators (``..``) MUST NOT be used. Tools MUST assume that license file content is valid UTF-8 encoded text, and SHOULD validate this and raise an error if it is not. -If the ``paths`` subkey is a non-empty array, build tools: - -- MUST treat each value as a verbatim, literal file path, and - MUST NOT treat them as glob patterns. - -- MUST include each listed file in all distribution archives. - -- MUST NOT match any additional license files beyond those explicitly - statically specified by the user under the ``paths`` subkey. +Literal paths (e.g. ``LICENSE``) are treated as valid globs which means they +can also be defined. -- MUST list each file path under a ``License-File`` field in the Core Metadata. +To achieve better portability, the filenames to match should only contain +the alphanumeric characters, underscores (``_``), hyphens (``-``) +and dots (``.``). -- MUST raise an error if one or more paths do not correspond to a valid file - in the project source that can be copied into the distribution archive. - -If the ``globs`` subkey is a non-empty array, build tools: +Build tools: - MUST treat each value as a glob pattern, and MUST raise an error if the pattern contains invalid glob syntax. -- MUST include all files matched by at least one listed pattern in all +- MUST include all files matched by a listed pattern in all distribution archives. -- MAY exclude files matched by glob patterns that can be unambiguously - determined to be backup, temporary, hidden, OS-generated or VCS-ignored. - - MUST list each matched file path under a ``License-File`` field in the Core Metadata. -- SHOULD issue a warning and MAY raise an error if no files are matched. - -- MAY issue a warning if any individual user-specified pattern +- MUST raise an error if any individual user-specified pattern does not match at least one file. -If the ``license-files`` key is present, and the ``paths`` or ``globs`` subkey +If the ``license-files`` key is present and is set to a value of an empty array, then tools MUST NOT include any license files and MUST NOT raise an error. -.. _639-default-patterns: - -If the ``license-files`` key is not present and not explicitly marked as -``dynamic``, tools MUST assume a default value of the following: - -.. code-block:: toml - - license-files.globs = ["LICEN[CS]E*", "COPYING*", "NOTICE*", "AUTHORS*"] - -In this case, tools MAY issue a warning if no license files are matched, -but MUST NOT raise an error. - -If the ``license-files`` key is marked as ``dynamic`` (and not present), -to preserve consistent behavior with current tools and help ensure the packages -they create are legally distributable, build tools SHOULD default to -including at least the license files matching the above patterns, unless the -user has explicitly specified their own. - Examples of valid license files declaration: .. code-block:: toml [project] - license-files = { globs = ["LICEN[CS]E*", "AUTHORS*"] } + license-files = ["LICEN[CS]E*", "AUTHORS*"] [project] - license-files.paths = ["licenses/LICENSE.MIT", "licenses/LICENSE.CC0"] + license-files = ["licenses/LICENSE.MIT", "licenses/LICENSE.CC0"] [project] - license-files = { paths = [] } + license-files = ["LICENSE.txt", "licenses/*"] [project] - license-files.globs = [] + license-files = [] Examples of invalid license files declaration: .. code-block:: toml [project] - license-files.globs = ["LICEN[CS]E*", "AUTHORS*"] - license-files.paths = ["LICENSE.MIT"] - -Reason: license-files.paths and license-files.globs are mutually exclusive. - -.. code-block:: toml - - [project] - license-files = { paths = ["..\LICENSE.MIT"] } + license-files = ["..\LICENSE.MIT"] Reason: ``..`` must not be used. ``\`` is an invalid path delimiter, ``/`` must be used. @@ -630,7 +584,7 @@ Reason: ``..`` must not be used. .. code-block:: toml [project] - license-files = { globs = ["LICEN{CSE*"] } + license-files = ["LICEN{CSE*"] Reason: "LICEN{CSE*" is not a valid glob. @@ -659,14 +613,9 @@ the ``license-files`` key instead. If the specified license ``file`` is present in the source tree, build tools SHOULD use it to fill the ``License-File`` field in the core metadata, and MUST include the specified file -as if it were specified in a ``license-file.paths`` field. +as if it were specified in a ``license-file`` field. If the file does not exist at the specified path, tools MUST raise an informative error as previously specified. -However, tools MUST also still assume the -:ref:`specified default value <639-default-patterns>` -for the ``license-files`` key and also include, -in addition to a license file specified under the ``license.file`` subkey, -any license files that match the specified list of patterns. Table values for the ``license`` key MAY be removed from a new version of the specification in a future PEP. @@ -751,9 +700,9 @@ and license classifiers retain backwards compatibility. A removal is left to a future PEP and a new version of the Core Metadata specification. Specification of the new ``License-File`` Core Metadata field and adding the -files in the distribution codifies the existing practices of many packaging -tools. It is designed to be largely backwards-compatible with their existing -use of that field. The new ``license-files`` key in the ``[project]`` table of +files in the distribution is designed to be largely backwards-compatible with +the existing use of that field in many packaging tools. +The new ``license-files`` key in the ``[project]`` table of ``pyproject.toml`` will only have an effect once users and tools adopt it. This PEP specifies that license files should be placed in a dedicated @@ -807,7 +756,8 @@ If an invalid ``License-Expression`` is used, the users will not be able to publish their package to PyPI and an error message will help them understand they need to use SPDX identifiers. It will be possible to generate a distribution with incorrect license metadata, -but not to publish one on PyPI or any other index server that enforces ``License-Expression`` validity. +but not to publish one on PyPI or any other index server that enforces +``License-Expression`` validity. For authors using the now-deprecated ``License`` field or license classifiers, packaging tools may warn them and inform them of the replacement, ``License-Expression``. diff --git a/peps/pep-0639/appendix-examples.rst b/peps/pep-0639/appendix-examples.rst index ed6d97165ea..68d8b5d7170 100644 --- a/peps/pep-0639/appendix-examples.rst +++ b/peps/pep-0639/appendix-examples.rst @@ -127,29 +127,30 @@ Putting it all together, our ``setup.cfg`` would be: setuptools/_vendor/packaging/LICENSE.APACHE setuptools/_vendor/packaging/LICENSE.BSD -In the ``[project]`` table of ``pyproject.toml``, with license files -specified explicitly via the ``paths`` subkey, this would look like: +In the ``[project]`` table of ``pyproject.toml``, license files +can be specified via glob patterns: .. code-block:: toml [project] license = "MIT AND (Apache-2.0 OR BSD-2-Clause)" - license-files.paths = [ - "LICENSE", - "setuptools/_vendor/LICENSE", - "setuptools/_vendor/LICENSE.APACHE", - "setuptools/_vendor/LICENSE.BSD", + license-files = [ + "LICENSE*", + "setuptools/_vendor/LICENSE*", ] -Or alternatively, matched via glob patterns, this could be: +Or alternatively, they can be specified explicitly (paths will be interpreted +as glob patterns): .. code-block:: toml [project] license = "MIT AND (Apache-2.0 OR BSD-2-Clause)" - license-files.globs = [ - "LICENSE*", - "setuptools/_vendor/LICENSE*", + license-files = [ + "LICENSE", + "setuptools/_vendor/LICENSE", + "setuptools/_vendor/LICENSE.APACHE", + "setuptools/_vendor/LICENSE.BSD", ] With either approach, the output Core Metadata in the distribution @@ -211,6 +212,7 @@ Some additional examples of valid ``License-Expression`` values: License-Expression: GPL-3.0-only WITH Classpath-Exception-2.0 OR BSD-3-Clause License-Expression: LicenseRef-Public-Domain OR CC0-1.0 OR Unlicense License-Expression: LicenseRef-Proprietary + License-Expression: LicenseRef-Custom-License .. _packaginglicense: https://github.com/pypa/packaging/blob/21.2/LICENSE diff --git a/peps/pep-0639/appendix-rejected-ideas.rst b/peps/pep-0639/appendix-rejected-ideas.rst index 990456c47b2..d66567cf0ac 100644 --- a/peps/pep-0639/appendix-rejected-ideas.rst +++ b/peps/pep-0639/appendix-rejected-ideas.rst @@ -149,66 +149,23 @@ Alternative possibilities related to the ``license`` key in the ``pyproject.toml`` project source metadata. -Add ``expression`` and ``files`` subkeys to table -''''''''''''''''''''''''''''''''''''''''''''''''' +Add new subkeys to table +'''''''''''''''''''''''' -A previous draft of PEP 639 added ``expression`` and ``files`` subkeys -to the existing ``license`` table in the project source metadata, to parallel -the existing ``file`` and ``text`` subkeys. While this seemed the -most obvious approach at first glance, it had serious drawbacks -relative to that ultimately taken here. - -This means two very different types of metadata are being -specified under the same top-level key that require very different handling, -and unlike the previous arrangement, the subkeys were not mutually -exclusive and could both be specified at once, with some subkeys potentially -being dynamic and others static, and mapping to different Core Metadata fields. - -There are further downsides to this as well. Both users and tools would need to -keep track of which fields are mutually exclusive with which of the others, -greatly increasing complexity, and the probability -of errors. Having so many different fields under the -same key leads to a much more complex mapping between -``[project]`` keys and Core Metadata fields, not in keeping with :pep:`621`. -This causes the ``[project]`` table naming and structure to diverge further -from both the Core Metadata and native formats of the various popular packaging -tools that use it. Finally, this results in the spec being significantly more -complex to understand and implement than the alternatives. - -The approach PEP 639 now takes, using the reserved top-level string value -of the ``license`` key, adding a new ``license-files`` key -and deprecating the ``license`` table subkeys (``text`` and ``file``), -avoids most of the issues identified above, -and results in a much clearer and cleaner design overall. -It allows ``license`` and ``license-files`` to be tagged -``dynamic`` independently, separates two independent types of metadata -(syntactically and semantically), restores a closer to 1:1 mapping of -``[project]`` table keys to Core Metadata fields, -and reduces nesting by a level for both. -Other than adding one extra key to the file, there was no significant -apparent downside to this latter approach, so it was adopted for PEP 639. - - -Add an ``expression`` subkey instead of a string value -'''''''''''''''''''''''''''''''''''''''''''''''''''''' +There were proposals to add various subkeys to the table. +Combining different types of metadata which require different handling, +adding new guidance regarding the subkeys mutual exclusivity and +the possibility to define some of them as dynamic would make the +transition harder and create more confusion rather than clarity for the users. +This approach has been rejected in favour of more flat ``pyproject.toml`` +design, clear mapping between ``pyproject.toml`` keys and Core Metadata fields, +and increased readability of the separate keys. + +Rejected proposals: -Adding just an ``expression`` subkey to the ``license`` table, -instead of using the top-level string value, -would be more explicit for readers and writers, -in line with PEP 639's goals. -However, it still has the downsides listed above -that are not specific to the inclusion of the ``files`` key. - -Relative to a flat string value, -it adds complexity and an extra level of nesting, -and requires users and tools to remember and handle -the mutual exclusivity of the subkeys -and remember which are deprecated, -instead of cleanly deprecating the table subkeys as a whole. -Furthermore, it is less clearly the "default" choice for modern use, -given users tend to gravitate toward the most obvious option. -Finally, it seems reasonable to follow the suggested guidance in :pep:`621`, -given the top-level string value was specifically reserved for this purpose. +- add ``expression`` and ``files`` subkeys to table +- add an ``expression`` subkey instead of a string value +- add a ``type`` key to treat ``text`` as expression Define a new top-level ``license-expression`` key @@ -267,39 +224,6 @@ Therefore, a top-level string value for ``license`` was adopted for PEP 639, as an earlier working draft had temporarily specified. -Add a ``type`` key to treat ``text`` as expression -'''''''''''''''''''''''''''''''''''''''''''''''''' - -Instead of using the reserved top-level string value -of the ``license`` key in the ``[project]`` table, -one could add a ``type`` subkey to the ``license`` table -to control whether ``text`` (or a string value) -is interpreted as free-text or a license expression. This could make -backward compatibility a bit easier, as older tools could ignore -it and always treat ``text`` as ``license``, while newer tools would -know to treat it as a license expression, if ``type`` was set appropriately. -Indeed, :pep:`621` seems to suggest something of this sort as a possible -way that SPDX license expressions could be implemented. - -However, it has got all the same downsides as in the previous item, -including greater complexity, a more complex mapping between the project -source metadata and Core Metadata and inconsistency between the presentation -in tool config, project source metadata and Core Metadata, -a harder deprecation, further bikeshedding over what to name it, -and inability to mark one but not the other as dynamic, among others. - -In addition, while theoretically a little easier in the short -term, in the long term it would mean users would always have to remember -to specify the correct ``type`` to ensure their license expression is -interpreted correctly, which adds work and potential for error; we could -never safely change the default while being confident that users -understand that what they are entering is unambiguously a license expression, -with all the false positive and false negative issues as above. - -Therefore, for these reasons, we reject this here in favor of -the reserved string value of the ``license`` key. - - Source metadata ``license-files`` key ------------------------------------- @@ -308,17 +232,30 @@ Alternatives considered for the ``license-files`` key in the path/glob type handling. -Add a ``type`` subkey to ``license-files`` -'''''''''''''''''''''''''''''''''''''''''' +Define mutually exclusve ``paths`` and ``globs`` subkeys to ``license-files`` +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +A previous draft of the PEP specified mutually exclusive ``paths`` and +``globs`` subkeys of the ``license-files`` ``[project]`` table key. +This was proposed to achieve the maximum clarity of the defined values for +both users and tools. +Allowing license files to be specified as literal paths would avoid edge cases, +such as those containing glob characters +(or those confusingly similar to them, as described in :pep:`672`). + +However, this approach introduces an extra level of nesting - in the very same +way that PEP 639 removes from the ``license`` key. This creates more burden +on project authors who need to disambiguate and choose one or the other +approach to specify their license files location. It was pointed out that +it is easily possible to incorrectly assume that paths also support +globs. -Instead of defining mutually exclusive ``paths`` and ``globs`` subkeys -of the ``license-files`` ``[project]`` table key, we could -achieve the same effect with a ``files`` subkey for the list and -a ``type`` subkey for how to interpret it. However, it offers no -real advantage in exchange for requiring more keystrokes, -increased complexity, as well as less flexibility in allowing both, -or another additional subkey in the future, as well as the need to bikeshed -over the subkey name. Therefore, it was rejected. +Therfore, it was decided against this approach in favor of a flat array value +which simplifies the specification and implementation, +and more closely matches the configuration format of existing tools. +The PEP recommends not to use other than alphanumerical symbols and dot +(``.``) in the filenames to not create confusion +when interpreting glob patterns. Only accept verbatim paths @@ -343,137 +280,25 @@ legal file, creating the package illegal to distribute. Tools can still determine the files to be included, based only on the glob patterns the user specified and the filenames in the package, without installing it, executing its code or even -examining its files. Furthermore, tools are explicitly allowed to warn -if specified glob patterns don't match any files. +examining its files. And, of course, sdists, wheels and others will have the full static list of files specified in their distribution metadata. -Perhaps most importantly, this would also exclude the currently specified -default value widely used by the most popular tools, and thus -be a major break to backward compatibility. -And of course, authors are welcome to specify their license -files explicitly via the ``paths`` table subkey, once they are aware of it and -find it suitable for their project. +Use a default value for ``license-files`` if not specified +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -Only accept glob patterns -''''''''''''''''''''''''' - -Conversely, all ``license-files`` strings could be treated as glob patterns. -This would slightly simplify the spec and implementation, avoid an extra level -of nesting, and more closely match the configuration format of existing tools. - -However, for the cost of a few characters, it ensures users are aware -whether they are entering globs or verbatim paths. Furthermore, allowing -license files to be specified as literal paths avoids edge cases, such as those -containing glob characters (or those confusingly or even maliciously similar -to them, as described in :pep:`672`). - -Including an explicit ``paths`` value ensures that the resulting -``License-File`` metadata is correct, complete and purely static in the -strictest sense of the term, with all license paths explicitly specified -in the ``pyproject.toml`` file, guaranteed to be included and with an early -error if any are missing. This is not practical to do, at least without -serious limitations for many workflows, if we must assume the items -are glob patterns rather than literal paths. - -This allows tools to locate them and know the exact values of the -``License-File`` Core Metadata fields without having to traverse the -source tree of the project and match globs, potentially allowing -more reliable programmatic inspection and processing. - -Therefore, given the relatively small cost and the significant benefits, -this approach was not adopted. - - -Infer whether paths or globs -'''''''''''''''''''''''''''' - -It was considered whether to simply allow specifying an array of strings -directly for the ``license-files`` key, rather than making it a table with -explicit ``paths`` and ``globs``. This would be simpler and avoid -an extra level of nesting, and more closely match the configuration format -of existing tools. However, it was ultimately rejected in favor of separate, -mutually exclusive ``paths`` and ``globs`` table subkeys. - -In practice, it only saves six extra characters in the ``pyproject.toml`` -(``license-files = [...]`` vs ``license-files.globs = [...]``), but allows -the user to explicitly declare their intent and serves as an unambiguous -indicator for tools to parse them as globs rather than verbatim paths. - -This, in turn, allows for clearly specified tool -behaviors for each case, many of which would be unreliable or impossible -without it and -behave more intuitively overall. These include, with ``paths``, -guaranteeing that each specified file is included and immediately -raising an error if one is missing, and with ``globs``, checking glob syntax, -excluding unwanted backup, temporary, or other such files, -and optionally warning if a glob doesn't match any files. -This also avoids edge cases (e.g. paths that contain glob characters) and -reliance on heuristics to determine interpretation. - - -.. _639-license-files-allow-flat-array: - -Also allow a flat array value -''''''''''''''''''''''''''''' - -Initially, after deciding to define ``license-files`` as a table of ``paths`` -and ``globs``, thought was given to making a top-level string array under the -``license-files`` key mean one or the other (probably ``globs``, to match most -current tools). This is slightly shorter, indicates to -the users which one is a preferred one, and allows a cleaner handling of -the empty case. - -However, this only saves six characters in the best case, and there -isn't an obvious choice. - -Flat may be better than nested, but in the face of ambiguity, users -may not resist the temptation to guess. Requiring users to explicitly specify -one or the other ensures they are aware of how their inputs will be handled, -and is more readable for others. It also makes -the spec and tool implementation slightly more complicated, and it can always -be added in the future, but not removed without breaking backward -compatibility. And finally, for the "preferred" option, it means there is -more than one obvious way to do it. - -Therefore, per :pep:`20`, the Zen of Python, this approach is rejected. - - -Allow both ``paths`` and ``globs`` subkeys -'''''''''''''''''''''''''''''''''''''''''' - -Allowing both ``paths`` and ``globs`` subkeys to be specified under the -``license-files`` table was considered, as it could potentially allow -more flexible handling for particularly complex projects. - -However, given the existing proposed approach already matches or exceeds the -capabilities of those offered in tools' config files, there isn't -clear demand for this, and it adds a large -amount of complexity in tool implementations and ``pyproject.toml`` -for relatively minimal gain. - -There would be many more edge cases to deal with, such as how to handle files -matched by both lists, and it conflicts with the current -specification for how tools should behave, such as when -no files match. - -Like the previous, if there is a clear need for it, it can be always allowed -in the future in a backward-compatible manner, -while the same is not true of disallowing it. -Therefore, it was decided to require the two subkeys to be mutually exclusive. - - -Rename ``paths`` subkey to ``files`` -'''''''''''''''''''''''''''''''''''' - -Initially, the name ``files`` was considered instead of the ``paths`` for the -subkey of ``license-files`` table. However, ``paths`` was ultimately -chosen to avoid duplication between -the table name (``license-files``) and the subkey name (``files``), i.e. -``license-files.files = ["LICENSE.txt"]``. It made it seem like -the preferred subkey when it was not, and didn't describe the format of the -string entry similarly to the existing ``globs``. +A previous draft of the PEP proposed a default value for detecting +license files in case the users have not declared any and not marked the key +as dynamic. +That value was defined as an array of globs: +``["LICEN[CS]E*", "COPYING*", "NOTICE*", "AUTHORS*"]`` + +However, this would create an exception among the existing metadata, +as no other key has got implicit defaults defined. Implicit values in +pyproject.toml keys are delegated to the ``dynamic`` field, +which is specified as being calculated. Also, the values were chosen +arbitrarily, without a strong justification why they should pose a standard. Must be marked dynamic to use defaults From 100a9e7365c6170283c196e27ef9953417f4969f Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 25 Jul 2024 14:56:44 -0700 Subject: [PATCH 22/27] PEP 715: clarify what `[package.tool]` is (#3873) --- peps/pep-0751.rst | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 5584d4bf268..a60811048b6 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -547,15 +547,21 @@ other is disallowed. ------------------ - Optional -- Same usage as that of the equivalent table from the - `pyproject.toml specification`_. +- Table +- Similar usage as that of the ``[tool]`` table from the + `pyproject.toml specification`_ , but at the package version level instead of + at the lock file level (which is also available via ``[tool]``). +- Useful for scoping package version/release details (e.g., recording signing + identities to then use to verify package integrity separately from where the + package is hosted, prototyping future extensions to this file format, etc.). ``[tool]`` ========== - Optional -- Same usage as that of the equivalent table from the +- Table +- Same usage as that of the equivalent ``[tool]`` table from the `pyproject.toml specification`_. From a1c4b12634eea3ea43bc9d61b938bb2e8313b154 Mon Sep 17 00:00:00 2001 From: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Fri, 26 Jul 2024 15:58:25 +0300 Subject: [PATCH 23/27] PEP 665: Superseded-By: 751 (#3875) --- peps/pep-0665.rst | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/peps/pep-0665.rst b/peps/pep-0665.rst index 923dc83d9d9..b66cc751000 100644 --- a/peps/pep-0665.rst +++ b/peps/pep-0665.rst @@ -8,9 +8,9 @@ Discussions-To: https://discuss.python.org/t/9911 Status: Rejected Type: Standards Track Topic: Packaging -Content-Type: text/x-rst Created: 29-Jul-2021 Post-History: 29-Jul-2021, 03-Nov-2021, 25-Nov-2021 +Superseded-By: 751 Resolution: https://discuss.python.org/t/pep-665-take-2-a-file-format-to-list-python-dependencies-for-reproducibility-of-an-application/11736/140 .. note:: @@ -1009,13 +1009,3 @@ CC0-1.0-Universal license, whichever is more permissive. .. _TOML: https://toml.io .. _version specifiers spec: https://packaging.python.org/specifications/version-specifiers/ .. _wheel file: https://packaging.python.org/specifications/binary-distribution-format/ - - -.. - Local Variables: - mode: indented-text - indent-tabs-mode: nil - sentence-end-double-space: t - fill-column: 70 - coding: utf-8 - End: From 52ad6864d0cf16bab03d08cadc3fb7e247bebb5b Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 26 Jul 2024 15:52:27 -0700 Subject: [PATCH 24/27] PEP 751: update based on feedback (#3877) * PEP 751: update based on feedback * Fix a section underline --- peps/pep-0751.rst | 314 ++++++++++++++++++++++++---------------------- 1 file changed, 165 insertions(+), 149 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index a60811048b6..a2d090ad63d 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -31,9 +31,9 @@ Currently, no standard exists to: - Create an immutable record, such as a lock file, of which dependencies were installed. -Considering there are at least four well-known solutions to this problem in the -community (``pip freeze``, pip-tools_, Poetry_, and PDM_), there seems to be an -appetite for lock files in general. +Considering there are at least five well-known solutions to this problem in the +community (``pip freeze``, pip-tools_, uv_, Poetry_, and PDM_), there seems to +be an appetite for lock files in general. Those tools also vary in what locking scenarios they support. For instance, ``pip freeze`` and pip-tools only generate lock files for the current @@ -63,11 +63,11 @@ used to create their lock file. The file format is designed to be human-readable. This is so that the contents of the file can be audited by a human to make sure no -undesired dependencies end up being included in the lock file. It is also to -facilitate easy understanding of what would be installed if the lock file -without necessitating running a tool, once again to help with auditing. Finally, -the format is designed so that viewing a diff of the file is easy by centralizing -relevant details. +undesired dependencies end up being included in the lock file. It is also +designed to facilitate easy understanding of what would be installed from the +lock file without necessitating running a tool, once again to help with +auditing. Finally, the format is designed so that viewing a diff of the file is +easy by centralizing relevant details. The file format is also designed to not require a resolver at install time. Being able to analyze dependencies in isolation from one another when listed in @@ -108,6 +108,11 @@ This is similar to what ``pip freeze`` and pip-tools_ support, but with more strictness of the exact files as well as incorporating support to specify the locked files for multiple environments in the same file. +Per-file locking should be used when the installation attempt should fail +outright if there is no explicitly pre-approved set of installation artifacts +for the target platform. For example: locking the deployment dependencies for a +managed web service. + Package Locking =============== @@ -133,6 +138,12 @@ what environments their contributors are working from. As already mentioned, this approach is supported by PDM_. Poetry_ has `shown some interest `__. +Per-package locking should be used when the exact set of potential target +platforms is not known when generating the lock file, as it allows installation +tools to choose the most appropriate artifacts for each platform from the +pre-approved set. For example: locking the development dependencies for an open +source project. + ============= Specification @@ -182,7 +193,7 @@ other is disallowed. - String - The name of the hash algorithm used for calculating all hash values. - Only a single hash algorithm is used for the entire file to allow the - ``[[package.files]]`` table to be written inline for readability and + ``[[packages.files]]`` table to be written inline for readability and compactness purposes by only listing a single hash value instead of multiple values based on multiple hash algorithms. - Specifying a single hash algorithm guarantees that an algorithm that the user @@ -205,30 +216,30 @@ other is disallowed. representing the direct, top-level dependencies to be installed. -``[[file-lock]]`` -================= +``[[file-locks]]`` +================== - Array of tables - Mutually exclusive with ``[package-lock]``. - The array's existence implies the use of the per-file locking approach. - An environment that meets all of the specified criteria in the table will be considered compatible with the environment that was locked for. -- Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be +- Lockers MUST NOT generate multiple ``[file-locks]`` tables which would be considered compatible for the same environment. - In instances where there would be a conflict but the lock is still desired, either separate lock files can be written or per-package locking can be used. -- Entries in array SHOULD be sorted by ``file-lock.name`` lexicographically. +- Entries in array SHOULD be sorted by ``file-locks.name`` lexicographically. -``file-lock.name`` ------------------- +``file-locks.name`` +------------------- - String - A unique name within the array for the environment this table represents. -``[file-lock.marker-values]`` ------------------------------ +``[file-locks.marker-values]`` +------------------------------ - Optional - Table of strings @@ -240,17 +251,18 @@ other is disallowed. updating the file. -``file-lock.wheel-tags`` ------------------------- +``file-locks.wheel-tags`` +------------------------- - Optional - Array of strings -- An unordered array of `wheel tags`_ which must be supported by the environment. +- An unordered array of `wheel tags`_ for which all tags must be supported by + the environment. - The array MAY not be exhaustive to allow for a smaller array as well as to - help prevent multiple ``[[file-lock]]`` tables being compatible with the + help prevent multiple ``[[file-locks]]`` tables being compatible with the same environment by having one array being a strict subset of another - ``file-lock.wheel-tags`` entry in the same file's - ``[[file-lock]]`` tables. + ``file-locks.wheel-tags`` entry in the same file's + ``[[file-locks]]`` tables. - Lockers SHOULD sort the keys lexicographically to minimize changes when updating the file. - Lockers MUST NOT include @@ -263,7 +275,7 @@ other is disallowed. ================== - Table -- Mutually exclusive with ``[[file-lock]]``. +- Mutually exclusive with ``[[file-locks]]``. - Signifies the use of the package locking approach. @@ -278,49 +290,49 @@ other is disallowed. same information. -``[[package]]`` -=============== +``[[packages]]`` +================ - Array of tables - The array contains all data on the locked package versions. -- Lockers SHOULD record packages in order by ``package.name`` lexicographically - and ``package.version`` by the sort order for `version specifiers`_. +- Lockers SHOULD record packages in order by ``packages.name`` lexicographically + and ``packages.version`` by the sort order for `version specifiers`_. - Lockers SHOULD record keys in the same order as written in this PEP to - minimmize changes when updating. + minimize changes when updating. - Designed so that relevant details as to why a package is included are in one place to make diff reading easier. -``package.name`` ----------------- +``packages.name`` +----------------- - String -- The `normalized name`_ of the package. +- The `normalized name`_ of the packages. - Part of what's required to uniquely identify this entry. -``package.version`` -------------------- +``packages.version`` +-------------------- - String -- The version of the package. +- The version of the packages. - Part of what's required to uniquely identify this entry. -``package.multiple-entries`` ----------------------------- +``packages.multiple-entries`` +----------------------------- - Boolean - If package locking via ``[package-lock]``, then the multiple entries for the - same package MUST be mutually exclusive via ``package.marker`` (this is not - required for per-file locking as the ``package.*.lock`` entries imply mutual + same package MUST be mutually exclusive via ``packages.marker`` (this is not + required for per-file locking as the ``packages.*.lock`` entries imply mutual exclusivity). - Aids in auditing by knowing that there are multiple entries for the same package that may need to be considered. -``package.description`` ------------------------ +``packages.description`` +------------------------ - Optional - String @@ -329,21 +341,21 @@ other is disallowed. purpose. -``package.simple-repo-package-url`` ------------------------------------ +``packages.simple-repo-package-url`` +------------------------------------ - Optional (although mutually exclusive with - ``package.files.simple-repo-package-url``) + ``packages.files.simple-repo-package-url``) - String - Stores the `project detail`_ URL from the `Simple Repository API`_. - Useful for generating Packaging URLs (aka PURLs). - When possible, lockers SHOULD include this or - ``package.files.simple-repo-package-url`` to assist with generating + ``packages.files.simple-repo-package-url`` to assist with generating `software bill of materials`_ (aka SBOMs). -``package.marker`` ------------------- +``packages.marker`` +------------------- - Optional - String @@ -354,8 +366,8 @@ other is disallowed. installed. -``package.requires-python`` ---------------------------- +``packages.requires-python`` +---------------------------- - Optional - String @@ -366,11 +378,11 @@ other is disallowed. ``package-lock.requires-python`` was chosen. - It should not provide useful information for installers as it would be captured by ``package-lock.requires-python`` and isn't relevant when - ``[[file-lock]]`` is used. + ``[[file-locks]]`` is used. -``package.dependents`` ----------------------- +``packages.dependents`` +----------------------- - Optional - Array of strings @@ -380,8 +392,8 @@ other is disallowed. - This does not provide information which influences installers. -``package.dependencies`` ------------------------- +``packages.dependencies`` +------------------------- - Optional - Array of strings @@ -389,62 +401,62 @@ other is disallowed. - Useful in analyzing why a package happens to be listed in the file for auditing purposes. - This does not provide information which influences the installer as - ``[[file-lock]]`` specifies the exact files to use and ``[package-lock]`` - applicability is determined by ``package.marker``. + ``[[file-locks]]`` specifies the exact files to use and ``[package-lock]`` + applicability is determined by ``packages.marker``. -``package.direct`` ------------------- +``packages.direct`` +------------------- - Optional (defaults to ``false``) - Boolean - Represents whether the installation is via a `direct URL reference`_. -``[[package.files]]`` ---------------------- +``[[packages.files]]`` +---------------------- -- Must be specified if ``[package.vcs]`` is not +- Must be specified if ``[packages.vcs]`` is not - Array of tables - Tables can be written inline. - Represents the files to potentially install for the package and version. -- Entries in ``[[package.files]]`` SHOULD be lexicographically sorted by - ``package.files.name`` key to minimze changes in diffs. +- Entries in ``[[packages.files]]`` SHOULD be lexicographically sorted by + ``packages.files.name`` key to minimze changes in diffs. -``package.files.name`` -'''''''''''''''''''''' +``packages.files.name`` +''''''''''''''''''''''' - String - The file name. - Necessary for installers to decide what to install when using package locking. -``package.files.lock`` -'''''''''''''''''''''' +``packages.files.lock`` +''''''''''''''''''''''' -- Required when ``[[file-lock]]`` is used +- Required when ``[[file-locks]]`` is used - Array of strings -- An array of ``file-lock.name`` values which signify that the file is to be - installed when the corresponding ``[[file-lock]]`` table applies to the +- An array of ``file-locks.name`` values which signify that the file is to be + installed when the corresponding ``[[file-locks]]`` table applies to the environment. -- There MUST only be a single file with any one ``file-lock.name`` entry per +- There MUST only be a single file with any one ``file-locks.name`` entry per package, regardless of version. -``package.files.simple-repo-package-url`` -''''''''''''''''''''''''''''''''''''''''' +``packages.files.simple-repo-package-url`` +'''''''''''''''''''''''''''''''''''''''''' - Optional (although mutually exclusive with - ``package.simple-repo-package-url``) + ``packages.simple-repo-package-url``) - String -- The value has the same meaning as ``package.simple-repo-package-url``. +- The value has the same meaning as ``packages.simple-repo-package-url``. - This key is available per-file to support :pep:`708` when some files override what's provided by another `Simple Repository API`_ index. -``package.files.origin`` -'''''''''''''''''''''''' +``packages.files.origin`` +''''''''''''''''''''''''' - Optional - String @@ -453,8 +465,8 @@ other is disallowed. for the file if not already downloaded/available. -``package.files.hash`` -'''''''''''''''''''''' +``packages.files.hash`` +''''''''''''''''''''''' - String - The hash value of the file contents using the hash algorithm specified by @@ -463,17 +475,17 @@ other is disallowed. with. -``[package.vcs]`` ------------------ +``[packages.vcs]`` +------------------ -- Must be specified if ``[[package.files]]`` is not (although may be specified - simultaneously with ``[[package.files]]``). +- Must be specified if ``[[packages.files]]`` is not (although may be specified + simultaneously with ``[[packages.files]]``). - Table representing the version control system containing the package and version. -``package.vcs.type`` -'''''''''''''''''''' +``packages.vcs.type`` +''''''''''''''''''''' - String - The type of version control system used. @@ -482,15 +494,15 @@ other is disallowed. of the direct URL data structure. -``package.vcs.origin`` -'''''''''''''''''''''' +``packages.vcs.origin`` +''''''''''''''''''''''' - String - The URI of where the repository was located when the lock file was generated. -``package.vcs.commit`` -'''''''''''''''''''''' +``packages.vcs.commit`` +''''''''''''''''''''''' - String - The commit ID for the repository which represents the package and version. @@ -498,53 +510,53 @@ other is disallowed. (e.g. no Git tags). -``package.vcs.lock`` -'''''''''''''''''''' +``packages.vcs.lock`` +''''''''''''''''''''' -- Required when ``[[file-lock]]`` is used +- Required when ``[[file-locks]]`` is used - An array of strings -- An array of ``file-lock.name`` values which signify that the repository at the - specified commit is to be installed when the corresponding ``[[file-lock]]`` +- An array of ``file-locks.name`` values which signify that the repository at the + specified commit is to be installed when the corresponding ``[[file-locks]]`` table applies to the environment. - A name in the array may only appear if no file listed in - ``package.files.lock`` contains the name for the same package, regardless of + ``packages.files.lock`` contains the name for the same package, regardless of version. -``package.directory`` ---------------------- +``packages.directory`` +---------------------- - Optional and only valid when ``[package-lock]`` is specified - String - A local directory where a source tree for the package and version exists. -- Not valid under ``[[file-lock]]`` as this PEP does not make an attempt to +- Not valid under ``[[file-locks]]`` as this PEP does not make an attempt to specify a mechanism for verifying file contents have not changed since locking was performed. -``[[package.build-requires]]`` ------------------------------- +``[[packages.build-requires]]`` +------------------------------- - Optional -- An array of tables whose structure matches that of ``[[package]]``. +- An array of tables whose structure matches that of ``[[packages]]``. - Each entry represents a package and version to use when building the enclosing package and version. -- The array is complete/locked like ``[[package]]`` itself (i.e. installers - follow the same installation procedure for ``[[package.build-requires]]`` as - ``[[package]]``) +- The array is complete/locked like ``[[packages]]`` itself (i.e. installers + follow the same installation procedure for ``[[packages.build-requires]]`` as + ``[[packages]]``) - Selection of which entries to use for an environment as the same as - ``[[package]]`` itself, albeit only applying when installing the build + ``[[packages]]`` itself, albeit only applying when installing the build back-end and its dependencies. - This helps with reproducibility of the building of a package by recording either what was or would have been used if the locker needed to build the - package. + packages. - If the installer and user choose to install from source and this array is missing then the installer MAY choose to resolve what to install for building at install time, otherwise the installer MUST raise an error. -``[package.tool]`` ------------------- +``[packages.tool]`` +------------------- - Optional - Table @@ -571,7 +583,7 @@ Expectations for Lockers - When creating a lock file for ``[package-lock]``, the locker SHOULD read the metadata of **all** files that end up being listed in - ``[[package.files]]`` to make sure all potential metadata cases are covered + ``[[packages.files]]`` to make sure all potential metadata cases are covered - If a locker chooses not to check every file for its metadata, the tool MUST either provide the user with the option to have all files checked (whether that is opt-in or out is left up to the tool), or the user is somehow notified @@ -580,9 +592,9 @@ Expectations for Lockers - Lockers MAY want to provide a way to let users provide the information necessary to install for multiple environments at once when doing per-file locking, e.g. supporting a JSON file format which specifies wheel tags and - marker values much like in ``[[file-lock]]`` for which multiple files can be + marker values much like in ``[[file-locks]]`` for which multiple files can be specified, which could then be directly recorded in the corresponding - ``[[file-lock]]`` table (if it allowed for unambiguous per-file locking + ``[[file-locks]]`` table (if it allowed for unambiguous per-file locking environment selection) .. code-block:: JSON @@ -598,13 +610,13 @@ Expectations for Installers --------------------------- - Installers MAY support installation of non-binary files - (i.e. source distributions, source trees, and VCS), but are not required to + (i.e. source distributions, source trees, and VCS), but are not required to. - Installers MUST provide a way to avoid non-binary file installation for - reproducibility and security purposes + reproducibility and security purposes. - Installers SHOULD make it opt-in to use non-binary file installation to - facilitate a secure-by-default approach + facilitate a secure-by-default approach. - Under per-file locking, if what to install is ambiguous then the installer - MUST raise an error + MUST raise an error. Installing for per-file locking @@ -612,27 +624,29 @@ Installing for per-file locking An example workflow is: -- Iterate through each ``[[file-lock]]`` table to find the one that applies to - the environment being installed for -- If no compatible environment is found an error MUST be raised -- If multiple environments are found to be compatible then an error MUST be raised -- For the compatible environment, iterate through each entry in ``[[package]]`` -- For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look - for any files with ``file-lock.name`` listed in ``package.files.lock`` +- Iterate through each ``[[file-locks]]`` table to find the one that applies to + the environment being installed for. +- If no compatible environment is found an error MUST be raised. +- If multiple environments are found to be compatible then an error MUST be + raised. +- For the compatible environment, iterate through each entry in + ``[[packages]]``. +- For each ``[[packages]]`` entry, iterate through ``[[packages.files]]`` to + look for any files with ``file-locks.name`` listed in ``packages.files.lock``. - If a file is found with a matching lock name, add it to the list of candidate - files to install and move on to the next ``[[package]]`` entry -- If no file is found then check if ``package.vcs.lock`` contains a match (no - match is also acceptable) -- If a ``[[package.files]]`` contains multiple matching entries an error MUST - be raised due to ambiguity for what is to be installed -- If multiple ``[[package]]`` entries for the same package have matching files - an error MUST be raised due to ambiguity for what is to be installed + files to install and move on to the next ``[[packages]]`` entry. +- If no file is found then check if ``packages.vcs.lock`` contains a match (no + match is also acceptable). +- If a ``[[packages.files]]`` contains multiple matching entries an error MUST + be raised due to ambiguity for what is to be installed. +- If multiple ``[[packages]]`` entries for the same package have matching files + an error MUST be raised due to ambiguity for what is to be installed. - Find and verify the candidate files and/or VCS entries based on their hash or - commit ID as appropriate + commit ID as appropriate. - If a source distribution or VCS was selected and - ``[[package.build-requires]]`` exists, then repeat the above process as - appropriate to install the build dependencies necessary to build the package -- Install the candidate files + ``[[packages.build-requires]]`` exists, then repeat the above process as + appropriate to install the build dependencies necessary to build the package. +- Install the candidate files. Installing for package locking @@ -641,22 +655,23 @@ Installing for package locking An example workflow is: - Verify that the environment is compatible with - ``package-lock.requires-python``; if it isn't an error MUST be raised -- Iterate through each entry in ``[package]]`` -- For each entry, if there's a ``package.marker`` key, evaluate the expression - - - If the expression is false, then move on - - Otherwise the package entry must be installed somehow -- Iterate through the files listed in ``[[package.files]]``, looking for the - "best" file to install -- If no file is found, check for ``[package.vcs]`` -- If no match is found, an error MUST be raised + ``package-lock.requires-python``; if it isn't an error MUST be raised. +- Iterate through each entry in ``[packages]]``. +- For each entry, if there's a ``packages.marker`` key, evaluate the expression. + + - If the expression is false, then move on. + - Otherwise the package entry must be installed somehow. + +- Iterate through the files listed in ``[[packages.files]]``, looking for the + "best" file to install. +- If no file is found, check for ``[packages.vcs]``. +- If no match is found, an error MUST be raised. - Find and verify the selected files and/or VCS entries based on their hash or - commit ID as appropriate + commit ID as appropriate. - If the match is a source distribution or VCS and - ``[[package.build-requires]]`` is provided, repeat the above as appropriate to - build the package -- Install the selected files + ``[[packages.build-requires]]`` is provided, repeat the above as appropriate + to build the package. +- Install the selected files. ======================= @@ -690,7 +705,7 @@ within the file in a ``[tool]`` entry or via a side channel external to the lock file itself. This PEP does not do anything to prevent a user from installing an incorrect -package. While including many details to help in auditing a package's inclusion, +packages. While including many details to help in auditing a package's inclusion, there isn't any mechanism to stop e.g. name confusion attacks via typosquatting. Lockers may be able to provide some UX to help with this (e.g. by providing download counts for a package). @@ -759,7 +774,7 @@ At one point, to handle the issue of metadata varying between files and thus require examining every released file for a package and version for accurate locking results, the idea was floated to introduce a new core metadata version which would require all metadata for all wheel files be the same for a single -version of a package. Ultimately, though, it was deemed unnecessary as this PEP +version of a packages. Ultimately, though, it was deemed unnecessary as this PEP will put pressure on people to make files consistent for performance reasons or to make indexes provide all the metadata separate from the wheel files themselves. As well, there's no easy enforcement mechanism, and so community @@ -938,5 +953,6 @@ CC0-1.0-Universal license, whichever is more permissive. .. _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ .. _software bill of materials: https://www.cisa.gov/sbom .. _TOML: https://toml.io/ +.. _uv: https://github.com/astral-sh/uv .. _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ .. _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ From 1770125792a00823f1b5a0b03e8126ff44f626b7 Mon Sep 17 00:00:00 2001 From: pauleveritt Date: Tue, 30 Jul 2024 16:10:09 -0400 Subject: [PATCH 25/27] Include three comments from @carreau (drop two bullets, *args in example, explain *args.) --- peps/pep-0750.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index a6f4ce0e1d4..9be4553238f 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -97,7 +97,7 @@ Let's start with a tag string which simply returns a static greeting: .. code-block:: python - def greet(): + def greet(*args): """Give a static greeting.""" return "Hello!" @@ -147,9 +147,7 @@ strings: - ``args[0]`` is still the string-like ``'Hello '``, this time with a trailing space - ``args[1]`` is an expression -- the ``{name}`` part -- Tag strings represent this part as an *interpolation* object -- An interpolation is a tuple whose first item is a lambda -- Calling this lambda evaluates the expression in the original scope where the tag string was defined +- Tag strings represent this part as an *interpolation* object as discussed below The ``*args`` list is a sequence of ``Decoded`` and ``Interpolation`` values. A "decoded" object is a string-like object with extra powers, as described below. An "interpolation" object is a @@ -261,8 +259,9 @@ Evaluating Tag Strings ---------------------- When the tag string is evaluated, the tag must have a binding, or a ``NameError`` -is raised; and it must be a callable, or a ``TypeError`` is raised. This behavior -follows from the de-sugaring of: +is raised; and it must be a callable, or a ``TypeError`` is raised. The callable +must accept a sequence of positional arguments. This behavior follows from the +de-sugaring of: .. code-block:: python From a7de80f639d17a534b58ab5ec9678184a24eb192 Mon Sep 17 00:00:00 2001 From: pauleveritt Date: Tue, 6 Aug 2024 14:37:37 -0400 Subject: [PATCH 26/27] From Carol, move the point about import to the following paragraph. --- peps/pep-0750.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index a5ae87b3104..24349de113f 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -736,12 +736,12 @@ Consumers can look at tag strings as starting from f-strings: - They look familiar - Scoping and syntax rules are the same -- You just need to import the tag function They first thing they need to absorb: unlike f-strings, the string isn't immediately evaluated "in-place". Something else (the tag function) happens. That's the second thing to teach: the tag functions do something particular. -Thus the concept of "domain specific languages" (DSLs). +Thus the concept of "domain specific languages" (DSLs). What's extra to +teach: you need to import the tag function before tagging a string. Tag function authors think in terms of making a DSL. They have business policies they want to provide in a Python-familiar way. With tag From 44868168bfb01927e6f4c4294736195307d44132 Mon Sep 17 00:00:00 2001 From: pauleveritt Date: Tue, 6 Aug 2024 14:39:59 -0400 Subject: [PATCH 27/27] Per Carol: Remove paragraph about lifecycles as that is about *a* DSL, not DSLs in general. --- peps/pep-0750.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/peps/pep-0750.rst b/peps/pep-0750.rst index 24349de113f..03dfc932c3d 100644 --- a/peps/pep-0750.rst +++ b/peps/pep-0750.rst @@ -751,10 +751,6 @@ the bar for making a DSL. Tag authors can begin with simple use cases. After authors gain experience, tag strings can be used to add larger patterns: lazy evaluation, intermediate representations, registries, and more. -Finally, framework authors can provide contact points with their lifecycles. -For example, decorators which tag function authors can use to memoize -interpolations in the function args. - Each of these points also match the teaching of decorators. In that case, a learner consumes something which applies to the code just after it. They don't need to know too much about decorator theory to take advantage of the