arrow-udf-python PYO3_PYTHON

[hanxuanliang]

Currently, I have introduced arrow-udf-python into databend and encountered this error.

run-command: PYO3_PYTHON=python3.12 cargo build

---

Env

arrow-udf-python = { package = "arrow-udf-python", git = "https://github.com/risingwavelabs/arrow-udf", rev = "23fe0dd" }

python version: python 3.12.2

BACKTRACE

error: failed to run custom build command for `arrow-udf-python v0.1.0 (https://github.com/risingwavelabs/arrow-udf?rev=23fe0dd#23fe0dd4)`

Caused by:
  process didn't exit successfully: `/Users/xxx/Desktop/code/contribute/databend/target/debug/build/arrow-udf-python-fa3a3e2a7b69d90f/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at /Users/xxx/.cargo/git/checkouts/arrow-udf-7ba7f51509153980/23fe0dd/arrow-udf-python/build.rs:9:5:
  arrow-udf-python requires Python 3.12 or later, but found 3.7
  hint: you can set `PYO3_PYTHON` environment varibale, e.g. `PYO3_PYTHON=python3.12`
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[hanxuanliang]

-- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.7
  shared=true
  abi3=true
  lib_name=python3.12
  lib_dir=/Users/xxx/.pyenv/versions/3.12.2/lib
  executable=/Users/xxx/.pyenv/versions/3.12.2/bin/python
  pointer_width=64
  build_flags=
  suppress_build_script_link_lines=false

[wangrunji0408]

I have no idea on this problem. Is it possible to be related with pyenv?

[sundy-li]

Can we use RustPython to interpret the Python scripts?

[wangrunji0408]

Can we use RustPython to interpret the Python scripts?

I have investigated RustPython. Its API documentation is far less complete than pyo3, making it harder to integrate. Considering the risk of its completeness and performance issues (someone said RustPython is generally 5-15x slower than CPython. ref), I did not choose RustPython as the runtime.

However, the major advantage of RustPython against pyo3 (CPython) is that natively it has no GIL limitation. Currently, to use CPython in a multithreaded environment, we have to switch between sub-interpreters, which has introduced significant performance overhead. I would be very appreciated if someone is willing to explore the RustPython solution. 😄