Skip to content

doc/languages-frameworks/python: Reword section to make commit rules a bit clearer #339643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FliegendeWurst merged 2 commits into from
Dec 9, 2024
Merged

doc/languages-frameworks/python: Reword section to make commit rules a bit clearer #339643

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3e2e5da
doc/languages-frameworks/python: Reword section to make commit rules …
Pandapip1 Sep 4, 2024
00a43d7
doc/languages-frameworks/python: update references to python 3.12
Pandapip1 Sep 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 29 additions & 31 deletions doc/languages-frameworks/python.section.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -506,9 +506,9 @@ are used in [`buildPythonPackage`](#buildpythonpackage-function).

Several versions of the Python interpreter are available on Nix, as well as a
high amount of packages. The attribute `python3` refers to the default
interpreter, which is currently CPython 3.11. The attribute `python` refers to
interpreter, which is currently CPython 3.12. The attribute `python` refers to
CPython 2.7 for backwards-compatibility. It is also possible to refer to
specific versions, e.g. `python311` refers to CPython 3.11, and `pypy` refers to
specific versions, e.g. `python312` refers to CPython 3.12, and `pypy` refers to
the default PyPy interpreter.

Python is used a lot, and in different ways. This affects also how it is
Expand All @@ -524,10 +524,10 @@ however, are in separate sets, with one set per interpreter version.
The interpreters have several common attributes. One of these attributes is
`pkgs`, which is a package set of Python libraries for this specific
interpreter. E.g., the `toolz` package corresponding to the default interpreter
is `python3.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`.
is `python3.pkgs.toolz`, and the CPython 3.12 version is `python312.pkgs.toolz`.
The main package set contains aliases to these package sets, e.g.
`pythonPackages` refers to `python.pkgs` and `python311Packages` to
`python311.pkgs`.
`pythonPackages` refers to `python.pkgs` and `python312Packages` to
`python312.pkgs`.

#### Installing Python and packages {#installing-python-and-packages}

Expand All @@ -552,7 +552,7 @@ with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#
executables are wrapped to be able to find each other and all of the modules.

In the following examples we will start by creating a simple, ad-hoc environment
with a nix-shell that has `numpy` and `toolz` in Python 3.11; then we will create
with a nix-shell that has `numpy` and `toolz` in Python 3.12; then we will create
a re-usable environment in a single-file Python script; then we will create a
full Python environment for development with this same environment.

Expand All @@ -568,18 +568,18 @@ temporary shell session with a Python and a *precise* list of packages (plus
their runtime dependencies), with no other Python packages in the Python
interpreter's scope.

To create a Python 3.11 session with `numpy` and `toolz` available, run:
To create a Python 3.12 session with `numpy` and `toolz` available, run:

```sh
$ nix-shell -p 'python311.withPackages(ps: with ps; [ numpy toolz ])'
$ nix-shell -p 'python312.withPackages(ps: with ps; [ numpy toolz ])'
```

By default `nix-shell` will start a `bash` session with this interpreter in our
`PATH`, so if we then run:

```Python console
[nix-shell:~/src/nixpkgs]$ python3
Python 3.11.3 (main, Apr 4 2023, 22:36:41) [GCC 12.2.0] on linux
Python 3.12.4 (main, Jun 6 2024, 18:26:44) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy; import toolz
```
Expand All @@ -599,12 +599,8 @@ will still get 1 wrapped Python interpreter. We can start the interpreter
directly like so:

```sh
$ nix-shell -p "python311.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
this derivation will be built:
/nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv
building '/nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv'...
created 273 symlinks in user environment
Python 3.11.2 (main, Feb 7 2023, 13:52:42) [GCC 12.2.0] on linux
$ nix-shell -p "python312.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
Python 3.12.4 (main, Jun 6 2024, 18:26:44) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>>
Expand Down Expand Up @@ -644,7 +640,7 @@ Executing this script requires a `python3` that has `numpy`. Using what we learn
in the previous section, we could startup a shell and just run it like so:

```ShellSession
$ nix-shell -p 'python311.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
$ nix-shell -p 'python312.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
The dot product of [1 2] and [3 4] is: 11
```

Expand Down Expand Up @@ -707,12 +703,12 @@ create a single script with Python dependencies, but in the course of normal
development we're usually working in an entire package repository.

As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file.
Say we want to have Python 3.11, `numpy` and `toolz`, like before,
Say we want to have Python 3.12, `numpy` and `toolz`, like before,
in an environment. We can add a `shell.nix` file describing our dependencies:

```nix
with import <nixpkgs> {};
(python311.withPackages (ps: with ps; [
(python312.withPackages (ps: with ps; [
numpy
toolz
])).env
Expand All @@ -729,7 +725,7 @@ What's happening here?
imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
brings all attributes of `nixpkgs` in the local scope. These attributes form
the main package set.
2. Then we create a Python 3.11 environment with the [`withPackages`](#python.withpackages-function) function, as before.
2. Then we create a Python 3.12 environment with the [`withPackages`](#python.withpackages-function) function, as before.
3. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument
that takes the set of all Python packages and returns a list of packages to
include in the environment. Here, we select the packages `numpy` and `toolz`
Expand All @@ -740,7 +736,7 @@ To combine this with `mkShell` you can:
```nix
with import <nixpkgs> {};
let
pythonEnv = python311.withPackages (ps: [
pythonEnv = python312.withPackages (ps: [
ps.numpy
ps.toolz
]);
Expand Down Expand Up @@ -894,8 +890,8 @@ information. The output of the function is a derivation.

An expression for `toolz` can be found in the Nixpkgs repository. As explained
in the introduction of this Python section, a derivation of `toolz` is available
for each interpreter version, e.g. `python311.pkgs.toolz` refers to the `toolz`
derivation corresponding to the CPython 3.11 interpreter.
for each interpreter version, e.g. `python312.pkgs.toolz` refers to the `toolz`
derivation corresponding to the CPython 3.12 interpreter.

The above example works when you're directly working on
`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
Expand All @@ -908,7 +904,7 @@ and adds it along with a `numpy` package to a Python environment.
with import <nixpkgs> {};

( let
my_toolz = python311.pkgs.buildPythonPackage rec {
my_toolz = python312.pkgs.buildPythonPackage rec {
pname = "toolz";
version = "0.10.0";
pyproject = true;
Expand All @@ -919,7 +915,7 @@ with import <nixpkgs> {};
};

build-system = [
python311.pkgs.setuptools
python312.pkgs.setuptools
];

# has no tests
Expand All @@ -932,15 +928,15 @@ with import <nixpkgs> {};
};
};

in python311.withPackages (ps: with ps; [
in python312.withPackages (ps: with ps; [
numpy
my_toolz
])
).env
```

Executing `nix-shell` will result in an environment in which you can use
Python 3.11 and the `toolz` package. As you can see we had to explicitly mention
Python 3.12 and the `toolz` package. As you can see we had to explicitly mention
for which Python version we want to build a package.

So, what did we do here? Well, we took the Nix expression that we used earlier
Expand Down Expand Up @@ -1946,7 +1942,7 @@ has security implications and is relevant for those using Python in a

When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1`
and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
and [PYTHONHASHSEED=0](https://docs.python.org/3.12/using/cmdline.html#envvar-PYTHONHASHSEED).
Both are also exported in `nix-shell`.

### How to provide automatic tests to Python packages? {#automatic-tests}
Expand Down Expand Up @@ -2017,10 +2013,12 @@ The following rules are desired to be respected:
* `meta.platforms` takes the default value in many cases.
It does not need to be set explicitly unless the package requires a specific platform.
* The file is formatted with `nixfmt-rfc-style`.
* Commit names of Python libraries should reflect that they are Python
libraries, so write for example `python311Packages.numpy: 1.11 -> 1.12`.
It is highly recommended to specify the current default version to enable
automatic build by ofborg.
* Commit names of Python libraries must reflect that they are Python
libraries (e.g. `python312Packages.numpy: 1.11 -> 1.12` rather than `numpy: 1.11 -> 1.12`).
* The current default version of python should be included
in commit messages to enable automatic builds by ofborg.
For example `python312Packages.numpy: 1.11 -> 1.12` should be used rather
than `python3Packages.numpy: 1.11 -> 1.12`.
Note that `pythonPackages` is an alias for `python27Packages`.
* Attribute names in `python-packages.nix` as well as `pname`s should match the
library's name on PyPI, but be normalized according to [PEP
Expand Down