Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Python Tutorial: Dependencies, PyPI, Order, Grammar #1313

Merged
merged 4 commits into from
May 2, 2024
Merged

docs: Python Tutorial: Dependencies, PyPI, Order, Grammar #1313

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
92566a6
docs: Python Tutorial: Dependencies, PyPI, Order, Grammar
JesperDramsch May 1, 2024
acfb410
docs: Markdown formatting and pre-commit
JesperDramsch May 1, 2024
73edda3
Update python.md
ruben-arts May 1, 2024
37534f4
Merge branch 'main' into patch-1
tdejager May 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
234 changes: 179 additions & 55 deletions docs/tutorials/python.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,24 @@
# Tutorial: Doing python development with pixi
# Tutorial: Doing Python development with Pixi

In this tutorial we will show you how to create a simple python project with pixi. We will show some of the features that pixi provides, that are currently
not a part of `pdm`, `poetry` etc.
In this tutorial, we will show you how to create a simple Python project with pixi.
We will show some of the features that pixi provides, that are currently not a part of `pdm`, `poetry` etc.

## Why is this useful?
Pixi builts upon on the conda ecosystem, which allows you to create a python environment with all the dependencies you need. This is especially useful when you are working with multiple python interpreters and bindings to C and C++ libraries. Let's give it a go! E.g GDAL from PyPI does not have the binary C dependencies, but the conda package does.

## pixi.toml and pyproject.toml
We support two manifests formats, `pyproject.toml` and `pixi.toml`. In this tutorial we will use the `pyproject.toml` format. Because it is the most common format for python projects.
Pixi builds upon the conda ecosystem, which allows you to create a Python environment with all the dependencies you need.
This is especially useful when you are working with multiple Python interpreters and bindings to C and C++ libraries.
For example, GDAL from PyPI does not have binary C dependencies, but the conda package does.
On the other hand, some packages are only available through PyPI, which `pixi` can also install for you.
Best of both world, let's give it a go!

## `pixi.toml` and `pyproject.toml`

We support two manifest formats: `pyproject.toml` and `pixi.toml`.
In this tutorial, we will use the `pyproject.toml` format because it is the most common format for Python projects.

## Let's get started

Lets start out by making a directory and creating a new `pyproject.toml` file.
Let's start out by making a directory and creating a new `pyproject.toml` file.

```shell
pixi init pixi_py --pyproject
Expand Down Expand Up @@ -42,7 +49,7 @@ pixi_py = { path = ".", editable = true }
[tool.pixi.tasks]
```

Let's add the python project to the tree:
Let's add the Python project to the tree:

=== "Linux & macOS"
```shell
Expand All @@ -67,50 +74,112 @@ We now have the following directory structure:
└── pyproject.toml
```

We've used a flat-layout here but we support both [flat- and src-layouts](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/#src-layout-vs-flat-layout).
We've used a flat-layout here but pixi supports both [flat- and src-layouts](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/#src-layout-vs-flat-layout).

### What's in the pyproject?
### What's in the `pyproject.toml`?

Okay, so let's have a look at what's sections have been added.
Okay, so let's have a look at what's sections have been added and how we can modify the `pyproject.toml`.

These things were added to the pyproject.toml file:
These first entries were added to the `pyproject.toml` file:

```toml
# 1. Main pixi entry
# Main pixi entry
[tool.pixi.project]
channels = ["conda-forge"]
# This is your machine platform by default
platforms = ["osx-arm64"]
```

The `channels` and `platforms` are added to the `[tool.pixi.project]` section.
Channels like `conda-forge` manage packages similar to PyPI but allow for different packages across languages.
The keyword `platforms` determines what platform the project supports.

The `pixi_py` package itself is added as an editable dependency.
This means that the package is installed in editable mode, so you can make changes to the package and see the changes reflected in the environment, without having to re-install the environment.

# 2. Editable installs
```toml
# Editable installs
[tool.pixi.pypi-dependencies]
pixi_py = { path = ".", editable = true }
```

Let's go over them one by one:
In pixi, unlike other package managers, this is explicitly stated in the `pyproject.toml` file.
The main reason being so that you can choose which environment this package should be included in.

### Managing both conda and PyPI dependencies in pixi

Our projects usually depend on other packages.

```shell
$ pixi add black
Added black
```

This will result in the following addition to the `pyproject.toml`:

1. The `channels` and `platforms` are added to the `[tool.pixi.project]` section. Channels like `conda-forge` manage packages similar to pypi but allow for different packages across languages. Platforms determines for what platform to generate the lock for.
2. The `pixi_py` package is added as an editable dependency. This means that the package is installed in editable mode, so you can make changes to the package and see the changes reflected in the environment. In pixi, unlike other package managers this is explicitly stated in the `pyproject.toml` file. The main reason being so that you can choose into which environment this package should be included.
```toml
# Dependencies
[tool.pixi.dependencies]
black = ">=24.4.2,<24.5"
```

Now let's add some optional dependencies:
But we can also be strict about the version that should be used with `pixi add black=24`, resulting in

```toml
[tool.pixi.dependencies]
black = "24.*"
```

Now, let's add some optional dependencies:

```shell
pixi add --pypi --feature test pytest
```

Which results in the following fields added to the `pyproject.toml`:
```toml
[project.optional-dependencies]
test = ["pytest"]
```

# 3. Environments
[tool.pixi.environments]
default = { solve-group = "default" }
test = { features = ["test"], solve-group = "default" }
After we have added the optional dependencies to the `pyproject.toml`, pixi automatically creates a [`feature`](../reference/configuration.md/#the-feature-and-environments-tables), which can contain a collection of `dependencies`, `tasks`, `channels`, and more.

Sometimes there are packages that aren't available on conda channels but are published on PyPI.
We can add these as well, which pixi will solve together with the default dependencies.

```shell
$ pixi add black --pypi
Added black
Added these as pypi-dependencies.
```

We've added the optional dependencies to the `pyproject.toml` and this automatically creates a [`feature`](../reference/configuration.md/#the-feature-and-environments-tables), which is a collection of `dependencies`, `tasks`, `channels` etc.
These features can be combined
which results in the addition to the `dependencies` key in the `pyproject.toml`

3. We've created the `default` and `test` environments. The `default` environment is the default environment that is created when you run `pixi install`. The `test` environment is created from the optional dependencies in the `pyproject.toml` file. You can execute commands in this environment with e.g `pixi run -e test python`
```toml
dependencies = ["black"]
```

??? note "Solve Groups"
Solve groups are a way to group dependencies together. This is useful when you have multiple environments that share the same dependencies. This way you can solve dependencies together. E.g maybe `pytest` is a dependency that influences the dependencies of the `default` environment. By putting these in the same solve group, you assure that the versions in `test` and `default` are exactly the same versions.
When using the `pypi-depenencies` you can make use of the `optional-dependencies` that other packages make available.
For example, `black` makes the `cli` dependencies option, which can be added with the `--pypi` keyword:

```shell
$ pixi add black[cli] --pypi
Added black[cli]
Added these as pypi-dependencies.
```

which updates the `dependencies` entry to

```toml
dependencies = ["black[cli]"]
```

??? note "Optional dependencies in `pixi.toml`"
This tutorial focuses on the use of the `pyproject.toml`, but in case you're curious, the `pixi.toml` would contain the following entry after the installation of a PyPI package including an optional dependency:
```toml
[pypi-dependencies]
black = { version = "*", extras = ["cli"] }
```


### Installation: `pixi install`
Expand All @@ -119,24 +188,27 @@ Now let's `install` the project with `pixi install`:

```shell
$ pixi install
✔ Project in /private/tmp/pixi_py is ready to use!
✔ Project in /path/to/pixi_py is ready to use!
```

We now have a new directory called `.pixi` in the project root. This directory contains the environment that was created when we ran `pixi install`. The environment is a conda environment that contains the dependencies that we specified in the `pyproject.toml` file. We can also install the test environment with `pixi install -e test`. We can use these environments for executing code.
We now have a new directory called `.pixi` in the project root.
This directory contains the environment that was created when we ran `pixi install`.
The environment is a conda environment that contains the dependencies that we specified in the `pyproject.toml` file.
We can also install the test environment with `pixi install -e test`.
We can use these environments for executing code.

We also have a new file called `pixi.lock` in the project root. This file contains the exact versions of the dependencies that were installed in the environment, across platforms.
We also have a new file called `pixi.lock` in the project root.
This file contains the exact versions of the dependencies that were installed in the environment across platforms.

## What's in the environment?

!!! Python interpreters
The python interpreter is also installed in the environment. This is because the python interpreter version is read from the `requires-python` field in the `pyproject.toml` file. This is used to determine the python version to install in the environment. This way pixi automatically manages/bootstraps the python interpreter for you, so no more `brew`, `apt` or other system install steps.

Using `pixi list` you can see what's in the environment, this is essentially a nicer view on the lock file:
Using `pixi list`, you can see what's in the environment, this is essentially a nicer view on the lock file:

```shell
$ pixi list
Package Version Build Size Kind Source
bzip2 1.0.8 h93a5062_5 119.5 KiB conda bzip2-1.0.8-h93a5062_5.conda
black 24.4.2 3.8 MiB pypi black-24.4.2-cp312-cp312-win_amd64.http.whl
ca-certificates 2024.2.2 hf0a4a13_0 152.1 KiB conda ca-certificates-2024.2.2-hf0a4a13_0.conda
libexpat 2.6.2 hebf3989_0 62.2 KiB conda libexpat-2.6.2-hebf3989_0.conda
libffi 3.4.2 h3422bc3_5 38.1 KiB conda libffi-3.4.2-h3422bc3_5.tar.bz2
Expand All @@ -152,11 +224,40 @@ pixi_py 0.1.0 pypi . (editable
xz 5.2.6 h57fd34a_0 230.2 KiB conda xz-5.2.6-h57fd34a_0.tar.bz2
```

Here you can see the different conda and pypi packages listed. And as you can see the `pixi_py` package, that we are working on, is installed in editable mode. Every environment in pixi is isolated, but re-uses files that are hard-linked from a central cache directory. This means that you can have multiple environments with the same packages, but only have the individual files stored once on disk.
!!! Python interpreters
The Python interpreter is also installed in the environment.
This is because the Python interpreter version is read from the `requires-python` field in the `pyproject.toml` file.
This is used to determine the Python version to install in the environment.
This way, pixi automatically manages/bootstraps the Python interpreter for you, so no more `brew`, `apt` or other system install steps.

Here, you can see the different conda and Pypi packages listed.
As you can see, the `pixi_py` package that we are working on is installed in editable mode.
Every environment in pixi is isolated but reuses files that are hard-linked from a central cache directory.
This means that you can have multiple environments with the same packages but only have the individual files stored once on disk.

We can create the `default` and `test` environments based on our own `test` feature from the `optional-dependency`:

```toml
# Environments
[tool.pixi.environments]
default = { solve-group = "default" }
test = { features = ["test"], solve-group = "default" }
```

??? note "Solve Groups"
Solve groups are a way to group dependencies together.
This is useful when you have multiple environments that share the same dependencies.
For example, maybe `pytest` is a dependency that influences the dependencies of the `default` environment.
By putting these in the same solve group, you ensure that the versions in `test` and `default` are exactly the same.

The `default` environment is created when you run `pixi install`.
The `test` environment is created from the optional dependencies in the `pyproject.toml` file.
You can execute commands in this environment with e.g. `pixi run -e test python`

## Getting code to run

Lets add some code to the `pixi_py` package. We will add a new function to the `pixi_py/__init__.py` file:
Let's add some code to the `pixi_py` package.
We will add a new function to the `pixi_py/__init__.py` file:

```python
from rich import print
Expand All @@ -167,22 +268,28 @@ def hello():
def say_hello():
print(*hello())
```
Now add the `rich` dependency using: `pixi add --pypi rich`.

Let's see if it this works by running:
Now add the `rich` dependency from PyPI using: `pixi add --pypi rich`.

Let's see if this works by running:

```shell
pixi r python -c "import pixi_py; pixi_py.say_hello()"
Hello, World! 🧛
```

??? note "Slow?"
The first time this might be slow because we install the project but the second time it will be instant.
This might be slow(2 minutes) the first time because pixi installs the project, but it will be near instant the second time.

So we are running a python interpreter installed from conda. And we are importing the `pixi_py` package that is installed in editable mode. And we are calling the `say_hello` function that we just added. And it works! Cool!
Pixi runs the self installed Python interpreter.
Then, we are importing the `pixi_py` package, which is installed in editable mode.
The code calls the `say_hello` function that we just added.
And it works! Cool!

## Testing this code

Okay, so let's add a test for this function. Let's add a `tests/test_me.py` file in root of the project.
Okay, so let's add a test for this function.
Let's add a `tests/test_me.py` file in the root of the project.

Giving us the following project structure:

Expand All @@ -209,10 +316,13 @@ $ pixi task add --feature test test "pytest"
✔ Added task `test`: pytest .
```

So pixi has a task system to make it easy to run commands. Similar to `npm` scripts or something you would specify in a `Justfile`.
So pixi has a task system to make it easy to run commands.
Similar to `npm` scripts or something you would specify in a `Justfile`.

??? tip "Pixi tasks"
Tasks are actually a pretty cool pixi feature that are powerful and are run in a cross-platform shell. You can do caching, dependencies and more. Read more about tasks in the [tasks](../features/advanced_tasks.md) section.
Tasks are actually a pretty cool pixi feature that is powerful and runs in a cross-platform shell.
You can do caching, dependencies and more.
Read more about tasks in the [tasks](../features/advanced_tasks.md) section.

```shell
$ pixi r test
Expand All @@ -228,11 +338,11 @@ test_me.py .
================================================================================================== 1 passed in 0.00s =================================================================================================
```

Cool it seems to be working!
Neat! It seems to be working!

### Test vs Default environment
Interesting thing is if we compare the output of
the two environments.

The interesting thing is if we compare the output of the two environments.

```shell
pixi list -e test
Expand All @@ -250,25 +360,38 @@ pytest 8.1.1 1.1 mib pypi pytest-8.1.
```

But the default environment is missing this package.
This way you can finetune your environments to only have the packages that are needed for that environment.
E.g you could also have a `dev` environment that has `pytest` and `ruff` installed, but you could omit these from the `prod` environment.
There is a [docker](https://github.com/prefix-dev/pixi/tree/main/examples/docker) example that shows how to set-up a minimal `prod` environment, and copy from there.
This way, you can finetune your environments to only have the packages that are needed for that environment.
E.g. you could also have a `dev` environment that has `pytest` and `ruff` installed, but you could omit these from the `prod` environment.
There is a [docker](https://github.com/prefix-dev/pixi/tree/main/examples/docker) example that shows how to set up a minimal `prod` environment and copy from there.

## Replacing PyPI packages with conda packages

## Replacing pypi packages with conda packages
Last thing, pixi provides the ability for `pypi` packages to depend on `conda` packages.
Let's add `pygments` to the `pyproject.toml` file. Which is a dependency of the `rich` package.
Let's confirm this with `pixi list`:

```shell
$ pixi list
Package Version Build Size Kind Source
...
pygments 2.17.2 4.1 MiB pypi pygments-2.17.2-py3-none-any.http.whl
...
```

Let's explicitly add `pygments` to the `pyproject.toml` file.
Which is a dependency of the `rich` package.

```shell
pixi add pygments
```

This will add the following to the `pyproject.toml` file:

```toml
[tool.pixi.dependencies]
pygments = ">=2.17.2,<2.18"
```

We can now see that the `pygments` package is now installed as conda package.
We can now see that the `pygments` package is now installed as a conda package.

```shell
$ pixi list
Expand All @@ -277,7 +400,7 @@ Package Version Build Size Kind Source
pygments 2.17.2 pyhd8ed1ab_0 840.3 KiB conda pygments-2.17.2-pyhd8ed1ab_0.conda
```

This way PyPI dependencies and conda dependencies can be mixed and matched, and seamlessly interoperate.
This way, PyPI dependencies and conda dependencies can be mixed and matched to seamlessly interoperate.

```shell
$ pixi r python -c "import pixi_py; pixi_py.say_hello()"
Expand All @@ -288,9 +411,10 @@ And it still works!

## Conclusion

In this tutorial you've seen how easy it is to use a `pyproject.toml` to manage your pixi dependencies. As well, as how to use PyPI and conda dependencies together in the same project.
In this tutorial, you've seen how easy it is to use a `pyproject.toml` to manage your pixi dependencies and environments.
We have also explored how to use PyPI and conda dependencies seamlessly together in the same project and install optional dependencies to manage Python packages.

Hopefully, this provides for flexible and powerful way to manage your python projects, and a fertile base for further exploration with pixi.
Hopefully, this provides a flexible and powerful way to manage your Python projects and a fertile base for further exploration with Pixi.

Thanks for reading! Happy Coding 🚀

Expand Down
Loading