Use Cython as numba alternative

[felixhekhorn]

related to #185 - for the moment I use this issue to save my work on a different place than just my computer ... eventually, I think, I can turn this into an PR.

at the moment it seems doing just poetry build is not enough, but doing just before a cythonize -X language_level=3 -a -i myfile.pyx seems to be sufficient

cc @scarrazza

Resources

• Add support for non pure python projects python-poetry/poetry#11
• https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules
• https://stackoverflow.com/questions/63679315/how-to-use-cython-with-poetry
• https://github.com/sdispater/pendulum/blob/master/build.py
• https://github.com/zoj613/htnorm/blob/main/build-wheels.sh
• https://github.com/FedericoStra/cython-package-example/blob/master/setup.py
• https://github.com/decred/tinydecred/blob/master/decred/build.py
• https://github.com/ADicksonLab/geomm/blob/v0.1.7/build.py
• https://pypi.org/project/cython-lint/

Files

`pyproject.toml`

[tool.poetry]
name = "pkgcy"
version = "0.0.0"
description = "Blubbes"
authors = ["Lorem Ipsum <[email protected]>"]
license = "MIT"
keywords = []

packages = [
    { include = "pkgcy" },
    # { include = "tests", format = "sdist" },
]
include = [
    # { path = "meson.build", format = "sdist" },
    # { path = "py.typed" },
    # C extensions must be included in the wheel distributions
    { path = "*.so", format = "wheel" },
    { path = "*.pyd", format = "wheel" }, # for Windows
]


[tool.poetry.dependencies]
python = "^3.8,<3.12"
numpy = "^1.24"
scipy = "^1.10.1"

[tool.poetry.group.dev.dependencies]
cython = ">=3.0.0b2"

[tool.poetry.build]
generate-setup-file = false
script = "mybuild.py" # the name build.py is taken by poetry itself


[build-system]
requires = ["poetry-core>=1.1.0a6", "Cython>=3.0.0b2"] # or >= 0.29?
build-backend = "poetry.core.masonry.api"

`pkgcy/__init__.py`

from scipy.integrate import quad
import os

if os.environ.get("BUILD_WHEEL",False):
    pass
else:
    import pyximport; pyximport.install()

from myfile import f as g

def f(x):
    return x

def run():
    return quad(g, 0.,1.)

`mybuild.py`

import os

# See if Cython is installed
try:
    from Cython.Build import cythonize
# Do nothing if Cython is not available
except ImportError:
    # Got to provide this function. Otherwise, poetry will fail
    def build(setup_kwargs):
        pass
# Cython is installed. Compile
else:
    # from setuptools import Extension
    # from setuptools.dist import Distribution
    from distutils.command.build_ext import build_ext

    # This function will be executed in setup.py:
    def build(setup_kwargs):
        # The file you want to compile
        extensions = [
            "myfile.pyx"
        ]

        # gcc arguments hack: enable optimizations
        os.environ['CFLAGS'] = '-O3'

        # Build
        setup_kwargs.update({
            'ext_modules': cythonize(
                extensions,
                language_level=3,
                compiler_directives={'linetrace': True},
            ),
            'cmdclass': {'build_ext': build_ext}
        })

`myfile.pyx`

# cython: language_level=3
def f(x):
    return x*x*x

[felixhekhorn]

Cython and numba:

• https://stackoverflow.com/questions/43016076/calling-cython-functions-from-numba-jitted-code
• https://numba.pydata.org/numba-doc/latest/extending/high-level.html#importing-cython-functions

Cython submodules:

• https://stackoverflow.com/questions/62245550/using-and-importing-cimporting-nested-packages-in-cython

myfilea.pyx

# cython: language_level=3

from src2.myfile2 import f2

cpdef api double foo(double x):
    return x * f2(x)

src2/myfile2.pyx

# cython: language_level=3
def f2(x):
    return 2.*x*x

pkgcy/__init__.py

from scipy.integrate import quad
import os

# if os.environ.get("BUILD_WHEEL",False):
#     pass
# else:
#     import pyximport; pyximport.install()

# from myfilea import f3a as g
# # import myfile

import numba as nb
import ctypes
from numba.extending import get_cython_function_address

addr = get_cython_function_address("myfilea", "foo")
functype = ctypes.CFUNCTYPE(ctypes.c_double, ctypes.c_double)
myexp = functype(addr)

@nb.njit
def f(x):
    return x * myexp(x)

def run():
    # print(myfile.__pyx_capi__)
    # return f(10.)
    return quad(f, 0.,1.)

and

• empty src2/__init__.pyd
• $ cythonize -X language_level=3 -f -a -i myfilea.pyx src2/myfile2.pyx

[alecandido]

We can call more or less in the same way any C library from Numba:

https://numba.readthedocs.io/en/stable/user/cfunc.html#calling-c-code-from-numba

At that point, I'd write the code directly in C/C++, or in Rust and providing a C API.
Moreover, providing the C library for external users was the goal since the beginning. If EKO consumes the same one, at least during the transition, it's not bad (eventually, when the top-level infrastructure will move out of Numba, we could also consume a different kind of API). In any case, we aim to drop Numba.

[felixhekhorn]

yes, but the main advantage of using Cython is the minimal effort in rewriting, because I'm still afraid of doing all in one gigantic step ... this way we could, maybe, do step by step, though I'm not sure still ...

[alecandido]

this way we could, maybe, do step by step, though I'm not sure still...

This is exactly what I was pointing out: the same feature that allows Cython to be introduced one step at a time could be used with whatever C library (because it's based on ctypes).

[felixhekhorn]

Cython is the minimal effort in rewriting

well, we would still need to rewrite everything from scratch (using C/Rust syntax)

[alecandido]

well, we would still need to rewrite everything from scratch (using C/Rust syntax)

For as long as it's just math, the C/Rust syntax is mostly Python syntax. It was Fortran in the first place...
(and doing it incrementally should make this a fully negligible concern)

[felixhekhorn]

closed in favour of #189