Custom build process -- support / timeline?

[dmontagu]

Cool project!

Is it currently possible to make use of custom build systems for use with cython- and/or cmake-built extensions? If not, is this on the roadmap?

This is something I need in many of my projects that I've gotten working well with poetry via the use of a build.py file. I found a TODO reference to custom build systems; rather than digging more deeply into the source code (especially as someone with minimal familiarity with Rust) or playing around with the tool too much, I figured I might get a quicker answer by just asking.

[David-OConnor]

This is the most notable missing feature. I haven't done the research on it yet beyond briefly skimming Flit's code, but it's something I need to add. I think mimicking flit would be a good starting point. (Or even wrapping it?) Currently, pyflow package and pyflow publish just wrap the normal setuptools/twine process.

Do you have a good example of a repo using systems like this I could experiment with?

[dmontagu]

I have based my pybind11/cmake examples on this: https://github.com/pybind/cmake_example/blob/master/setup.py

I based cython-only with poetry on the implementation described in this comment: python-poetry/poetry#11 (comment) (the referenced pendulum project).

In another project, I was able to combine the two above to work with poetry:

Click to expand

"""
Adapted from https://github.com/pybind/cmake_example
"""
import os
import platform
import re
import subprocess
import sys
import sysconfig
from distutils.version import LooseVersion
from typing import Any, Dict

import Cython.Build
from numpy import get_include as get_numpy_include
from setuptools.command.build_ext import build_ext
from setuptools.extension import Extension


class CMakeExtension(Extension):
    name: str  # exists, even though IDE doesn't find it

    def __init__(self, name: str, sourcedir: str="") -> None:
        super().__init__(name, sources=[])
        self.sourcedir = os.path.abspath(sourcedir)


class ExtensionBuilder(build_ext):
    def run(self) -> None:
        self.validate_cmake()
        super().run()

    def build_extension(self, ext: Extension) -> None:
        if isinstance(ext, CMakeExtension):
            self.build_cmake_extension(ext)
        else:
            super().build_extension(ext)

    def validate_cmake(self) -> None:
        cmake_extensions = [x for x in self.extensions if isinstance(x, CMakeExtension)]
        if len(cmake_extensions) > 0:
            try:
                out = subprocess.check_output(["cmake", "--version"])
            except OSError:
                raise RuntimeError(
                    "CMake must be installed to build the following extensions: "
                    + ", ".join(e.name for e in cmake_extensions)
                )
            if platform.system() == "Windows":
                cmake_version = LooseVersion(re.search(r"version\s*([\d.]+)", out.decode()).group(1))  # type: ignore
                if cmake_version < "3.1.0":
                    raise RuntimeError("CMake >= 3.1.0 is required on Windows")

    def build_cmake_extension(self, ext: CMakeExtension) -> None:
        extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.name)))
        cmake_args = ["-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=" + extdir, "-DPYTHON_EXECUTABLE=" + sys.executable]

        cfg = "Debug" if self.debug else "Release"
        # cfg = 'Debug'
        build_args = ["--config", cfg]

        if platform.system() == "Windows":
            cmake_args += ["-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{}={}".format(cfg.upper(), extdir)]
            if sys.maxsize > 2 ** 32:
                cmake_args += ["-A", "x64"]
            build_args += ["--", "/m"]
        else:
            cmake_args += ["-DCMAKE_BUILD_TYPE=" + cfg]
            build_args += ["--", "-j4"]
        cmake_args += ["-DPYTHON_INCLUDE_DIR={}".format(sysconfig.get_path("include"))]

        env = os.environ.copy()
        env["CXXFLAGS"] = '{} -DVERSION_INFO=\\"{}\\"'.format(env.get("CXXFLAGS", ""), self.distribution.get_version())
        if not os.path.exists(self.build_temp):
            os.makedirs(self.build_temp)
        subprocess.check_call(["cmake", ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
        subprocess.check_call(["cmake", "--build", "."] + build_args, cwd=self.build_temp)

def build(setup_kwargs: Dict[str, Any]) -> None:
    cython_modules = Cython.Build.cythonize([
        Extension(
            "cython_module",
            sources=["cython_module.pyx"],
            include_dirs=[get_numpy_include(), "."],
        )
    ])
    cmake_modules = [CMakeExtension("pybind_module.compiled", sourcedir="pybind_module/cxx")]
    ext_modules = cython_modules + cmake_modules
    setup_kwargs.update(
        {
            "ext_modules": ext_modules,
            "cmdclass": dict(build_ext=ExtensionBuilder),
            "zip_safe": False,
        }
    )

Sorry I don't have anything public I can share, but hopefully the pendulum repo and pybind11 example are enough?

[David-OConnor]

That's what I need; thank you.

[David-OConnor]

The latest commit now supports a build parameter in pyproject.toml that points to a python file to run prior to executing the generated setup.py, as well as some fixes to parsing poetry's pyproject.toml metatdata: It now appears to build pendulum and lets me install/use the built wheel, but I'm not sure how to verify the c extension was built correctly. Are you able to build pyflow from source from the repo to test if that works on your project? If not, I'll build a binary. Note that all the build logic must be included in whatever build points to, eg build.py, since setup.py is never executed by design.

The build is still handled by setuptools run through the venv Python interpreter, but I think this is an ok solution for now, if it works.