Skip to content

[BUILD] Use build-system requires to install pybind11 #4450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ThomasRaoux merged 1 commit into from
Aug 30, 2024
Merged

[BUILD] Use build-system requires to install pybind11 #4450

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 4 additions & 10 deletions .github/workflows/integration-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,6 @@ jobs:
id: cache-key
run: |
echo "llvm=$(cat cmake/llvm-hash.txt | cut -c 1-8)" >> $GITHUB_OUTPUT
echo "pybind11=$(cat cmake/pybind11-version.txt)" >> $GITHUB_OUTPUT
echo "nvidia=$(cat cmake/nvidia-toolchain-version.txt)" >> $GITHUB_OUTPUT
echo "json=$(cat cmake/json-version.txt)" >> $GITHUB_OUTPUT
echo "datetime=$(date -u -Iseconds)" >> $GITHUB_OUTPUT
Expand All @@ -179,9 +178,8 @@ jobs:
path: |
~/.triton/llvm
~/.triton/nvidia
~/.triton/pybind11
~/.triton/json
key: ${{ runner.os }}-${{ runner.arch }}-llvm-${{ steps.cache-key.outputs.llvm }}-nvidia-${{ steps.cache-key.outputs.nvidia }}-pybind11-${{ steps.cache-key.outputs.pybind11 }}-json-${{ steps.cache-key.outputs.json }}
key: ${{ runner.os }}-${{ runner.arch }}-llvm-${{ steps.cache-key.outputs.llvm }}-nvidia-${{ steps.cache-key.outputs.nvidia }}-json-${{ steps.cache-key.outputs.json }}
- # Cache ~/.triton/cache because the vast majority of unit test time is
# spent compiling. Triton won't (well, should not) use these cached files
# if something internal to Triton changes, because Triton's internal
Expand Down Expand Up @@ -309,7 +307,6 @@ jobs:
id: cache-key
run: |
echo "llvm=$(cat cmake/llvm-hash.txt | cut -c 1-8)" >> $GITHUB_OUTPUT
echo "pybind11=$(cat cmake/pybind11-version.txt)" >> $GITHUB_OUTPUT
echo "nvidia=$(cat cmake/nvidia-toolchain-version.txt)" >> $GITHUB_OUTPUT
echo "json=$(cat cmake/json-version.txt)" >> $GITHUB_OUTPUT
echo "datetime=$(date -u -Iseconds)" >> $GITHUB_OUTPUT
Expand All @@ -322,9 +319,8 @@ jobs:
path: |
~/.triton/llvm
~/.triton/nvidia
~/.triton/pybind11
~/.triton/json
key: ${{ runner.os }}-${{ runner.arch }}-llvm-${{ steps.cache-key.outputs.llvm }}-nvidia-${{ steps.cache-key.outputs.nvidia }}-pybind11-${{ steps.cache-key.outputs.pybind11 }}-json-${{ steps.cache-key.outputs.json }}
key: ${{ runner.os }}-${{ runner.arch }}-llvm-${{ steps.cache-key.outputs.llvm }}-nvidia-${{ steps.cache-key.outputs.nvidia }}-json-${{ steps.cache-key.outputs.json }}
- # Cache ~/.triton/cache because the vast majority of unit test time is
# spent compiling. Triton won't (well, should not) use these cached files
# if something internal to Triton changes, because Triton's internal
Expand Down Expand Up @@ -442,7 +438,6 @@ jobs:
id: cache-key
run: |
echo "llvm=$(cat cmake/llvm-hash.txt | cut -c 1-8)" >> $GITHUB_OUTPUT
echo "pybind11=$(cat cmake/pybind11-version.txt)" >> $GITHUB_OUTPUT
echo "nvidia=$(cat cmake/nvidia-toolchain-version.txt)" >> $GITHUB_OUTPUT
echo "json=$(cat cmake/json-version.txt)" >> $GITHUB_OUTPUT
echo "datetime=$(date -u -Iseconds)" >> $GITHUB_OUTPUT
Expand All @@ -455,9 +450,8 @@ jobs:
path: |
~/.triton/llvm
~/.triton/nvidia
~/.triton/pybind11
~/.triton/json
key: ${{ runner.os }}-${{ runner.arch }}-llvm-${{ steps.cache-key.outputs.llvm }}-nvidia-${{ steps.cache-key.outputs.nvidia }}-pybind11-${{ steps.cache-key.outputs.pybind11 }}-json-${{ steps.cache-key.outputs.json }}
key: ${{ runner.os }}-${{ runner.arch }}-llvm-${{ steps.cache-key.outputs.llvm }}-nvidia-${{ steps.cache-key.outputs.nvidia }}-json-${{ steps.cache-key.outputs.json }}
- # Cache ~/.triton/cache because the vast majority of unit test time is
# spent compiling. Triton won't (well, should not) use these cached files
# if something internal to Triton changes, because Triton's internal
Expand Down Expand Up @@ -494,7 +488,7 @@ jobs:
python3 -m venv ~/.venv
source ~/.venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install cython setuptools wheel cmake==3.24 ninja pytest-xdist lit
python3 -m pip install cython setuptools wheel cmake==3.24 ninja pytest-xdist lit pybind11
- name: Install Triton
env:
TRITON_BUILD_WITH_CCACHE: "true"
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/integration-tests.yml.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,6 @@ jobs:
id: cache-key
run: |
echo "llvm=$(cat cmake/llvm-hash.txt | cut -c 1-8)" >> $GITHUB_OUTPUT
echo "pybind11=$(cat cmake/pybind11-version.txt)" >> $GITHUB_OUTPUT
echo "nvidia=$(cat cmake/nvidia-toolchain-version.txt)" >> $GITHUB_OUTPUT
echo "json=$(cat cmake/json-version.txt)" >> $GITHUB_OUTPUT
echo "datetime=$(date -u -Iseconds)" >> $GITHUB_OUTPUT
Expand All @@ -204,9 +203,8 @@ jobs:
path: |
~/.triton/llvm
~/.triton/nvidia
~/.triton/pybind11
~/.triton/json
key: ${{ runner.os }}-${{ runner.arch }}-llvm-${{ steps.cache-key.outputs.llvm }}-nvidia-${{ steps.cache-key.outputs.nvidia }}-pybind11-${{ steps.cache-key.outputs.pybind11 }}-json-${{ steps.cache-key.outputs.json }}
key: ${{ runner.os }}-${{ runner.arch }}-llvm-${{ steps.cache-key.outputs.llvm }}-nvidia-${{ steps.cache-key.outputs.nvidia }}-json-${{ steps.cache-key.outputs.json }}

# Cache ~/.triton/cache because the vast majority of unit test time is
# spent compiling. Triton won't (well, should not) use these cached files
Expand Down Expand Up @@ -440,7 +438,7 @@ jobs:
python3 -m venv ~/.venv
source ~/.venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install cython setuptools wheel cmake==3.24 ninja pytest-xdist lit
python3 -m pip install cython setuptools wheel cmake==3.24 ninja pytest-xdist lit pybind11
- name: Install Triton
env:
TRITON_BUILD_WITH_CCACHE: "true"
Expand Down
1 change: 0 additions & 1 deletion cmake/pybind11-version.txt
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

[build-system]
requires = ["setuptools>=40.8.0", "wheel", "cmake>=3.18", "ninja>=1.11.1"]
requires = ["setuptools>=40.8.0", "wheel", "cmake>=3.18", "ninja>=1.11.1", "pybind11>=2.13.1"]

# We're incrementally switching from autopep8 to ruff.
[tool.autopep8]
Expand Down
26 changes: 14 additions & 12 deletions python/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
from setuptools.command.egg_info import egg_info
from wheel.bdist_wheel import bdist_wheel

import pybind11


@dataclass
class Backend:
Expand Down Expand Up @@ -142,16 +144,6 @@ class Package(NamedTuple):
syspath_var_name: str


# pybind11
def get_pybind11_package_info():
pybind11_version_path = os.path.join(get_base_dir(), "cmake", "pybind11-version.txt")
with open(pybind11_version_path, "r") as pybind11_version_file:
version = pybind11_version_file.read().strip()
name = f"pybind11-{version}"
url = f"https://github.com/pybind/pybind11/archive/refs/tags/v{version}.tar.gz"
return Package("pybind11", name, url, "PYBIND11_INCLUDE_DIR", "", "PYBIND11_SYSPATH")


# json
def get_json_package_info():
url = "https://github.com/nlohmann/json/releases/download/v3.11.3/include.zip"
Expand Down Expand Up @@ -365,8 +357,17 @@ def run(self):
for ext in self.extensions:
self.build_extension(ext)

def get_pybind11_cmake_args(self):
pybind11_sys_path = get_env_with_keys(["PYBIND11_SYSPATH"])
if pybind11_sys_path:
pybind11_include_dir = os.path.join(pybind11_sys_path, "include")
else:
pybind11_include_dir = pybind11.get_include()
return [f"-DPYBIND11_INCLUDE_DIR={pybind11_include_dir}"]

def get_proton_cmake_args(self):
cmake_args = get_thirdparty_packages([get_json_package_info(), get_pybind11_package_info()])
cmake_args = get_thirdparty_packages([get_json_package_info()])
cmake_args += self.get_pybind11_cmake_args()
cupti_include_dir = get_env_with_keys(["TRITON_CUPTI_PATH"])
if cupti_include_dir == "":
cupti_include_dir = os.path.join(get_base_dir(), "third_party", "nvidia", "backend", "include")
Expand All @@ -381,7 +382,8 @@ def build_extension(self, ext):
lit_dir = shutil.which('lit')
ninja_dir = shutil.which('ninja')
# lit is used by the test suite
thirdparty_cmake_args = get_thirdparty_packages([get_pybind11_package_info(), get_llvm_package_info()])
thirdparty_cmake_args = get_thirdparty_packages([get_llvm_package_info()])
thirdparty_cmake_args += self.get_pybind11_cmake_args()
extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.path)))
# create build directories
if not os.path.exists(self.build_temp):
Expand Down