Improved Rust py example

python/examples/README.md

@@ -1,7 +1,7 @@
 This folder offers various examples of how to use Python in wasm modules.
 
  - [./basic](./basic/) is a collection of small snippets that demonstrate how to run `python.wasm` from the command line or via [Docker](https://docs.docker.com/get-docker/)
- - [./embedding/wasi-command-rs](./embedding/) shows how one can embed the static wasm32-wasi `libpython` into a command Wasm module using Rust.
+ - [./embedding/wasi-py-rs](./embedding/) shows how one can embed the static wasm32-wasi `libpython` into a command Wasm module using Rust.
  - [./bindings](./bindings/) is a sample application that demonstrates how one can use host-to-python and python-to-host bindings
 
  Note: `build.rs` does not have any download or performance optimization - the necessary `wasm32-wasi` dependencies are fetched and unpacked on each build run.

python/examples/embedding/wasi-py-rs/Cargo.toml

@@ -0,0 +1,11 @@
+[package]
+name = "wasi-py-rs"
+version = "0.1.0"
+edition = "2021"
+
+[dependencies]
+pyo3 = { version = "0.19.0", features = ["abi3-py311"] }
+wlr-libpy = { git = "https://github.com/vmware-labs/webassembly-language-runtimes.git", branch="rust-py-example", features = ["py_main"] }
+
+[build-dependencies]
+wlr-libpy = { git = "https://github.com/vmware-labs/webassembly-language-runtimes.git", branch="rust-py-example", features = ["build"] }

python/examples/embedding/wasi-py-rs/README.md

@@ -0,0 +1,91 @@
+# About
+
+Example that embeds CPython via libpython into a Wasm module written in Rust.
+
+Offers a couple of WASI Command (exporting `_start`) Wasm modules, written in Rust and demonstrates interaction with simple Python code via [pyo3](https://pyo3.rs/v0.19.0/).
+
+# How to run
+
+Make sure you have `cargo` with the `wasm32-wasi` target. For running we use `wasmtime`, but the module will work with any WASI-compliant runtime.
+
+Just run `./run_me.sh` in the current folder. You will see something like this
+
+```
+wlr/python/examples/embedding/wasi-py-rs $$ ./run_me.sh
+   Compiling pyo3-build-config v0.18.3
+   ...
+    Finished dev [unoptimized + debuginfo] target(s) in 26.43s
+
+Calling a WASI Command which embeds Python (adding a custom module implemented in Rust) and calls a custom function:
++ wasmtime --mapdir /usr::target/wasm32-wasi/wasi-deps/usr target/wasm32-wasi/debug/py-func-caller.wasm
+Hello from Python (libpython3.11.a / 3.11.3 (tags/v3.11.3:f3909b8, Apr 28 2023, 09:45:45) [Clang 15.0.7 ]) in Wasm(Rust).
+args= (('John', 21, ['male', 'student']), ('Jane', 22, ['female', 'student']), ('George', 75, ['male', 'retired']))
+Original people: [Person(Name: "John", Age: 21, Tags:["male", "student"]), Person(Name: "Jane", Age: 22, Tags:["female", "student"]), Person(Name: "George", Age: 75, Tags:["male", "retired"])]
+Filtered people by `student`: [Person(Name: "John", Age: 21, Tags:["male", "student"]), Person(Name: "Jane", Age: 22, Tags:["female", "student"])]
++ set +x
+
+35mCalling a WASI Command which wraps the Python binary (adding a custom module implemented in Rust):
++ wasmtime --mapdir /usr::target/wasm32-wasi/wasi-deps/usr target/wasm32-wasi/debug/py-wrapper.wasm -- -c 'import person as p; pp = [p.Person("a", 1), p.Person("b", 2)]; pp[0].add_tag("X"); print("Filtered: ", p.filter_by_tag(pp, "X"))'
+Filtered:  [Person(Name: "a", Age: 1, Tags:["X"])]
++ set +x
+```
+
+# About the code
+
+To see how you can expand this example start with `pyo3`'s documentation on [calling Python from Rust](https://pyo3.rs/v0.18.3/python_from_rust).
+
+The main purpose here is to show how to configure the build and dependencies.
+
+The code has the following structure:
+
+```
+src
+├── bin
+│   ├── py-func-caller.rs  -  A WASI command which defines and calls a Python function that uses the "person" module.
+│   └── py-wrapper.rs  -  A wrapper around Py_Main, which adds "person" as a built-in module.
+├── lib.rs  -  A library that allows one to call a Python function (passed as text) with arbitrary Rust Tuple arguments.
+└── py_module.rs  -  A simple Python module ("person") implemented in Rust. Offers creation and tagging of people via the person.Person class.
+```
+
+# Build and dependencies
+
+For pyo3 to work the final binary needs to link to `libpython3.11.a`. The WLR project provides a pre-build `libpython` static library (based on [wasi-sdk](https://github.com/WebAssembly/wasi-sdk)), which depends on `wasi-sdk`. To setup the build properly you will need to provide several static libs and configure the linker to use them properly.
+
+We provide a helper crate [wlr-libpy](../../../tools/wlr-libpy/), which can be used to fetch the pre-built libpython.
+
+Take a look at [Cargo.toml](./Cargo.toml) to see how to add it as a build dependency:
+
+```toml
+[build-dependencies]
+wlr-libpy = { git = "https://github.com/vmware-labs/webassembly-language-runtimes.git", branch="rust-py-example", features = ["build"] }
+```
+
+Then, in the [build.rs](./build.rs) file we only need to add this to the `main` method:
+
+```rs
+    use wlr_libpy::bld_cfg::configure_static_libs;
+    configure_static_libs().unwrap().emit_link_flags();
+```
+
+This will ensure that all required libraries are downloaded and the linker is configured to use them.
+
+Here is a diagram of the relevant dependencies
+
+```mermaid
+graph LR
+    wasi_py_rs["wasi-py-rs"] --> wlr_libpy["wlr-libpy"]
+    wlr_libpy --> wlr_assets["wlr-assets"]
+    wlr_assets --> wasi_sysroot["wasi-sysroot-19.0.tar.gz"]
+    wlr_assets --> clang_builtins["libclang_rt.builtins-wasm32-wasi-19.0.tar.gz"]
+    wlr_assets --> libpython["libpython-3.11.3-wasi-sdk-19.0.tar.gz"]
+
+    wasi_sysroot --> libwasi-emulated-signal.a
+    wasi_sysroot --> libwasi-emulated-getpid.a
+    wasi_sysroot --> libwasi-emulated-process-clocks.a
+    clang_builtins --> libclang_rt.builtins-wasm32.a
+    libpython --> libpython3.11.a
+
+    wasi_py_rs["wasi-py-rs"] --> pyo3["pyo3"]
+    pyo3 --> pyo3-ffi
+    pyo3-ffi -..-> libpython3.11.a
+```

python/examples/embedding/wasi-py-rs/build.rs

@@ -0,0 +1,4 @@
+fn main() {
+    use wlr_libpy::bld_cfg::configure_static_libs;
+    configure_static_libs().unwrap().emit_link_flags();
+}

python/examples/embedding/wasi-py-rs/run_me.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+
+if ! (rustup target list | grep "installed" | grep -q "wasm32-wasi"); then
+    echo "Please run 'rustup target add wasm32-wasi' first"
+    exit 1
+fi
+
+set -e
+
+PYO3_NO_PYTHON=1 cargo build --target=wasm32-wasi
+echo -e "\n\033[35mCalling a WASI Command which embeds Python (adding a custom module implemented in Rust) and calls a custom function:\033[0m"
+set -x
+wasmtime \
+    --mapdir /usr::target/wasm32-wasi/wasi-deps/usr \
+    target/wasm32-wasi/debug/py-func-caller.wasm
+set +x
+
+echo -e "\n\033[35mCalling a WASI Command which wraps the Python binary (adding a custom module implemented in Rust):\033[0m"
+set -x
+wasmtime \
+    --mapdir /usr::target/wasm32-wasi/wasi-deps/usr \
+    target/wasm32-wasi/debug/py-wrapper.wasm \
+    -- -c \
+    'import person as p; pp = [p.Person("a", 1), p.Person("b", 2)]; pp[0].add_tag("X"); print("Filtered: ", p.filter_by_tag(pp, "X"))'
+set +x

python/examples/embedding/wasi-py-rs/src/bin/py-func-caller.rs

@@ -0,0 +1,35 @@
+use pyo3::{append_to_inittab, PyResult};
+
+use wasi_py_rs::call_function;
+use wasi_py_rs::py_module::make_person_module;
+
+pub fn main() -> PyResult<()> {
+    append_to_inittab!(make_person_module);
+
+    let function_code = "def my_func(*args, **kwargs):
+    import sys
+    print(f'Hello from Python (libpython3.11.a / {sys.version}) in Wasm(Rust).\\nargs=', args)
+
+    import person
+    people = []
+    for name, age, tags in args:
+        p = person.Person(name, age)
+        for t in tags:
+            p.add_tag(t)
+        people.append(p)
+
+    filtered = person.filter_by_tag(people, 'student')
+    print(f'Original people: {people}')
+    print(f'Filtered people by `student`: {filtered}')
+    ";
+
+    call_function(
+        "my_func",
+        function_code,
+        (
+            ("John", 21, ["male", "student"]),
+            ("Jane", 22, ["female", "student"]),
+            ("George", 75, ["male", "retired"]),
+        ),
+    )
+}

python/examples/embedding/wasi-py-rs/src/bin/py-wrapper.rs

@@ -0,0 +1,9 @@
+use pyo3::append_to_inittab;
+use wasi_py_rs::py_module::make_person_module;
+use wlr_libpy::py_main::py_main;
+
+pub fn main() {
+    append_to_inittab!(make_person_module);
+
+    py_main(std::env::args().collect());
+}

python/examples/embedding/wasi-py-rs/src/lib.rs

@@ -0,0 +1,24 @@
+use pyo3::types::{PyModule, PyTuple};
+use pyo3::{IntoPy, Py, PyAny, PyResult, Python};
+
+pub mod py_module;
+use py_module::make_person_module;
+
+pub fn call_function<T: IntoPy<Py<PyTuple>>>(
+    function_name: &str,
+    function_code: &str,
+    args: T,
+) -> PyResult<()> {
+    pyo3::append_to_inittab!(make_person_module);
+
+    pyo3::prepare_freethreaded_python();
+
+    Python::with_gil(|py| -> PyResult<()> {
+        let fun: Py<PyAny> = PyModule::from_code(py, function_code, "", "")?
+            .getattr(function_name)?
+            .into();
+
+        fun.call1(py, args)?;
+        Ok(())
+    })
+}