Convert Python examples to proper packages (#5966)

### What

This PR does the following:
- Convert all Python examples to proper python packages, replacing the
`requirements.txt` file with a `pyproject.toml` file.
- Applies same conversion to `tests/python/test_api`.
- Add WIP support for running example with pixi
- Add a `all_examples` meta-project.
- Fix `dev-tools build-examples` for the new example format.
- Add a `dev-tools build-examples install` command to `pip install` the
examples for the given channel (prerequisite to `build-examples rrd`).
- Fix CI for the new example format.
- Updates pixi to 0.19.1 in CI.

#### pixi support

This WIP pixi support allows running examples in the `examples`
environment:

```
pixi run -e examples clock
```

Note that by default pixi builds the environment with a shim/empty/fake
rerun package (from `examples/python/_empty_rerun_sdk`), which is a
work-around to allow manual installation of rerun in the environment. As
a result, at first (and whenever a rebuild is needed), this must be run:

```
pixi run -e examples py-build
```

#### `all_examples`

This is a meta-package that dynamically depends on all examples. The
original hope was to be able to depend on it from pixi, avoiding the
need to list example explicitly. However, this is current impossible for
a number a reason:

- doing so means that examples would not be installed as editable,
causing issue with `__file__` relative paths
(#6054)
- doing so we leak full path in the lock file
- pixi currently has an issue where indirect dependency with git
dependencies are broken

`all_examples` also has a very basic CLI tool that allows listing all
examples in a format suitable for pixi: `all_examples list`.

Also, it should be relatively easy to cleanly reimplement the
functionality of `run_all.py`, which is currently broken:
- #6057

As it stands, it allows for efficiently checking dependency conflicts,
especially with uv:
```shell
cd /tmp
uv venv
source .venv/bin/activate
uv pip install -e path/to/rerun/examples/python/all_examples   # ok??
```

#### TODO

- [x] convert a few example to python package
- [x] initial version of a `all_examples` meta-package
- [x] fix re_build_example
- [x] **RE-ENABLE LINK CHECKS!!!!**
- ~~properly working pixi setup~~
- ~~python version matrix testing setup~~

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/5966?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/5966?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!

- [PR Build Summary](https://build.rerun.io/pr/5966)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.

.github/workflows/auto_release_crates.yml

@@ -26,7 +26,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Update crate versions
         run: pixi run python scripts/ci/crates.py version --bump prerelease

.github/workflows/contrib_checks.yml

@@ -51,7 +51,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Python format check
         run: pixi run py-fmt-check
@@ -92,7 +92,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Codegen check
         run: pixi run codegen --force --check
@@ -114,7 +114,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Rerun lints
         run: pixi run lint-rerun
@@ -159,7 +159,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
           environments: cpp
 
       # TODO(emilk): make this work somehow. Right now this just results in

.github/workflows/contrib_rerun_py.yml

@@ -57,7 +57,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
           environments: wheel-test
 
       - name: Build Wheel

.github/workflows/on_pull_request_target_contrib.yml

@@ -29,7 +29,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Update PR description
         run: |

.github/workflows/release.yml

@@ -84,7 +84,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Update crate versions
         id: versioning
@@ -379,7 +379,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Commit new version
         run: |

.github/workflows/reusable_build_and_upload_rerun_cli.yml

@@ -164,7 +164,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Build web-viewer (release)
         run: pixi run rerun-build-web-release

.github/workflows/reusable_build_and_upload_wheels.yml

@@ -189,7 +189,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Get sha
         id: get-sha

.github/workflows/reusable_build_examples.yml

@@ -64,7 +64,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
           environments: wheel-test
 
       - name: Download Wheel
@@ -75,9 +75,8 @@ jobs:
 
       - name: Install example dependencies
         run: |
-          pixi run -e wheel-test pip install \
-            -r scripts/ci/requirements-examples-${{ inputs.CHANNEL }}.txt \
-            --no-input
+          pixi run -e wheel-test build-examples install \
+            --channel ${{ inputs.CHANNEL }}
 
       - name: Install built wheel
         run: |

.github/workflows/reusable_build_js.yml

@@ -66,7 +66,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Install yarn dependencies
         run: pixi run yarn --cwd rerun_js install

.github/workflows/reusable_build_web.yml

@@ -66,7 +66,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Build web-viewer (release)
         run: |

.github/workflows/reusable_checks.yml

@@ -86,7 +86,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Codegen check
         run: |
@@ -104,7 +104,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Set up Python
         uses: actions/setup-python@v5
@@ -156,7 +156,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: prettier --check
         run: pixi run misc-fmt-check

.github/workflows/reusable_checks_cpp.yml

@@ -80,7 +80,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
           environments: cpp
 
       - name: Set up Rust

.github/workflows/reusable_checks_python.yml

@@ -35,7 +35,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Python format check
         run: pixi run py-fmt-check

.github/workflows/reusable_checks_rust.yml

@@ -63,7 +63,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       # We need to build the web viewer for `rust_checks.py` to succeed.
       # We build in release so that we can reuse the results for actual publishing, if necessary

.github/workflows/reusable_deploy_docs.yml

@@ -203,7 +203,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Doxygen C++ docs
         run: pixi run cpp-docs

.github/workflows/reusable_publish_js.yml

@@ -65,7 +65,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Publish packages
         env:

.github/workflows/reusable_publish_web.yml

@@ -76,7 +76,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
           environments: wheel-test
 
       # built by `reusable_build_and_publish_wheels`

.github/workflows/reusable_publish_wheels.yml

@@ -155,7 +155,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - id: "auth"
         uses: google-github-actions/auth@v2

.github/workflows/reusable_release_crates.yml

@@ -30,7 +30,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
 
       - name: Build web-viewer (release)
         run: pixi run rerun-build-web-release

.github/workflows/reusable_run_notebook.yml

@@ -39,7 +39,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
           environments: wheel-test
 
       - name: Download Wheel

.github/workflows/reusable_test_wheels.yml

@@ -147,7 +147,7 @@ jobs:
 
       - uses: prefix-dev/[email protected]
         with:
-          pixi-version: v0.19.0
+          pixi-version: v0.19.1
           environments: wheel-test
 
       - name: Download Wheel

.gitignore

@@ -1,7 +1,7 @@
 .DS_Store
 
 # Codegen stuff:
-crates/re_types/source_hash.txt
+crates/re_types/source_hash.txt§
 crates/re_types_builder/source_hash.txt
 
 # C++ and CMake stuff:
@@ -27,6 +27,7 @@ _deps
 
 # Python virtual environment:
 **/venv*
+**/.venv*
 .python-version
 
 # Python build artifacts:

.mypy.ini

@@ -1,6 +1,6 @@
 [mypy]
 files = rerun_py/rerun_sdk/rerun, rerun_py/tests, examples/python
-exclude = examples/python/objectron/proto|examples/python/ros_node
+exclude = examples/python/objectron|examples/python/ros_node
 namespace_packages = True
 show_error_codes = True
 strict = True

crates/re_dev_tools/src/build_examples/example.rs

@@ -11,6 +11,7 @@ use anyhow::Context;
 pub struct Example {
     pub name: String,
     pub title: String,
+    pub dir: PathBuf,
     pub description: String,
     pub tags: Vec<String>,
     pub thumbnail_url: String,
@@ -60,21 +61,35 @@ impl Example {
         else {
             anyhow::bail!("example {name:?} has no frontmatter");
         };
+        let script_path = dir.join(language.entrypoint_path(&dir, name)?);
         Ok(Some(Example {
             name: name.to_owned(),
             title: readme.title,
+            dir,
             description: readme.description,
             tags: readme.tags,
             thumbnail_url: readme.thumbnail,
             thumbnail_dimensions: readme.thumbnail_dimensions,
-            script_path: dir.join(language.entrypoint_path()),
+            script_path,
             script_args: readme.build_args,
             readme_body: body,
             language,
         }))
     }
 }
 
+#[derive(Debug)]
+pub struct CouldNotFindEntryPoint(pub PathBuf);
+
+impl std::error::Error for CouldNotFindEntryPoint {}
+
+impl Display for CouldNotFindEntryPoint {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.write_str("could not find entrypoint for example: ")?;
+        self.0.display().fmt(f)
+    }
+}
+
 #[derive(Clone, Copy)]
 pub enum Language {
     Rust,
@@ -112,12 +127,32 @@ impl Language {
     /// For example:
     /// - `main.py` for Python
     /// - `src/main.rs` for Rust
-    pub fn entrypoint_path(&self) -> &'static Path {
+    pub fn entrypoint_path(
+        &self,
+        example_dir: &Path,
+        example_name: &str,
+    ) -> anyhow::Result<PathBuf> {
         match self {
-            Language::Rust => Path::new("src/main.rs"),
-            Language::Python => Path::new("main.py"),
-            Language::C => Path::new("main.c"),
-            Language::Cpp => Path::new("main.cpp"),
+            Language::Rust => Ok(PathBuf::from("src/main.rs")),
+            Language::Python => {
+                // we must handle two cases:
+                // - single script named after the example: `example.py`
+                // - package named after the example: `example/`
+
+                let mut script_path = PathBuf::from(example_name);
+                if example_dir.join(&script_path).exists() {
+                    return Ok(script_path);
+                }
+
+                script_path.set_extension("py");
+                if example_dir.join(&script_path).exists() {
+                    return Ok(script_path);
+                }
+
+                anyhow::bail!(CouldNotFindEntryPoint(example_dir.into()))
+            }
+            Language::C => Ok(PathBuf::from("main.c")),
+            Language::Cpp => Ok(PathBuf::from("main.cpp")),
         }
     }
 }
@@ -168,7 +203,7 @@ pub enum Channel {
     /// Examples built for each release, plus all `Main` examples.
     Release,
 
-    /// Examples built nightly, plus all `Main` and `Nightly`.
+    /// Examples built nightly, plus all `Main` and `Release`.
     Nightly,
 }
 
@@ -235,14 +270,17 @@ impl Channel {
                 }
 
                 eprintln!("{name:?}: added");
+                let dir = folder.path();
+                let script_path = dir.join(language.entrypoint_path(&dir, &name)?);
                 examples.push(Example {
                     name,
                     title: readme.title,
+                    dir,
                     description: readme.description,
                     tags: readme.tags,
                     thumbnail_url: readme.thumbnail,
                     thumbnail_dimensions: readme.thumbnail_dimensions,
-                    script_path: folder.path().join(language.entrypoint_path()),
+                    script_path,
                     script_args: readme.build_args,
                     readme_body: body,
                     language: Language::Python,