Skip to content

Include CUDA enabled pyarrow #146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xhochy merged 14 commits into from
May 28, 2020
Merged

Include CUDA enabled pyarrow #146

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
2bf71bc
Include pyarrow
pearu May 20, 2020
877ba52
MNT: Re-rendered with conda-build 3.19.2, conda-smithy 3.7.3, and con…
pearu May 20, 2020
8fe1859
Skip building pyarrow for aarch64
pearu May 20, 2020
6061ee1
Use azure provider for linux_aarch64
pearu May 20, 2020
4a0c756
MNT: Re-rendered with conda-build 3.19.2, conda-smithy 3.7.3, and con…
pearu May 20, 2020
7d17b71
Re-enable aarch64 for pyarrow
pearu May 20, 2020
1e346ad
Try fixing -march flag for aarch64
pearu May 21, 2020
86d85e1
Switch linux_aarch64 back to default.
pearu May 21, 2020
0e79569
MNT: Re-rendered with conda-build 3.19.2, conda-smithy 3.7.3, and con…
pearu May 21, 2020
1eb263e
Fix ARROW_ARMV8_ARCH usage.
pearu May 21, 2020
080819f
Fix the boost-cpp comment location.
pearu May 25, 2020
d9b9f82
Remove boost-cpp comment as obsolete.
pearu May 25, 2020
a633dc4
Rename pyarrow to arrow-py.
pearu May 26, 2020
aff54f7
Build pyarrow.gandiva unconditionally. Use pin_subpackage.. instead o…
pearu May 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions .ci_support/migrations/grpc_cpp129.yaml
View file
Open in desktop

This file was deleted.

7 changes: 0 additions & 7 deletions .ci_support/migrations/re2202005.yaml
View file
Open in desktop

This file was deleted.

5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 31 additions & 0 deletions recipe/bld-pyarrow.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
pushd "%SRC_DIR%"\python

@rem the symlinks for cmake modules don't work here
del cmake_modules\BuildUtils.cmake
del cmake_modules\SetupCxxFlags.cmake
del cmake_modules\CompilerInfo.cmake
del cmake_modules\FindNumPy.cmake
del cmake_modules\FindPythonLibsNew.cmake
copy /Y "%SRC_DIR%\cpp\cmake_modules\BuildUtils.cmake" cmake_modules\
copy /Y "%SRC_DIR%\cpp\cmake_modules\SetupCxxFlags.cmake" cmake_modules\
copy /Y "%SRC_DIR%\cpp\cmake_modules\CompilerInfo.cmake" cmake_modules\
copy /Y "%SRC_DIR%\cpp\cmake_modules\FindNumPy.cmake" cmake_modules\
copy /Y "%SRC_DIR%\cpp\cmake_modules\FindPythonLibsNew.cmake" cmake_modules\

SET ARROW_HOME=%LIBRARY_PREFIX%
SET SETUPTOOLS_SCM_PRETEND_VERSION=%PKG_VERSION%
SET PYARROW_BUILD_TYPE=release
SET PYARROW_WITH_S3=1
SET PYARROW_WITH_HDFS=1
SET PYARROW_WITH_DATASET=1
SET PYARROW_WITH_FLIGHT=1
SET PYARROW_WITH_GANDIVA=1
SET PYARROW_WITH_PARQUET=1
SET PYARROW_CMAKE_GENERATOR=Ninja

%PYTHON% setup.py ^
build_ext ^
install --single-version-externally-managed ^
--record=record.txt
Comment on lines +26 to +29
Copy link
Member

@jakirkham jakirkham May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question about pip here

if errorlevel 1 exit 1
popd
41 changes: 41 additions & 0 deletions recipe/build-pyarrow.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/bin/sh

set -e
set -x

# Build dependencies
export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
export PYARROW_WITH_HDFS=1
export PYARROW_WITH_ORC=1
export PYARROW_WITH_PARQUET=1
export PYARROW_WITH_PLASMA=1
export PYARROW_WITH_S3=1
BUILD_EXT_FLAGS=""

# Enable CUDA support
if [[ ! -z "${cuda_compiler_version+x}" && "${cuda_compiler_version}" != "None" ]]
then
export PYARROW_WITH_CUDA=1
BUILD_EXT_FLAGS="${BUILD_EXT_FLAGS} --with-cuda"
else
export PYARROW_WITH_CUDA=0
fi

# Resolve: Make Error at cmake_modules/SetupCxxFlags.cmake:338 (message): Unsupported arch flag: -march=.
if [[ "$(uname -m)" = "aarch64" ]]
then
export PYARROW_CMAKE_OPTIONS="-DARROW_ARMV8_ARCH=armv8-a"
fi

cd python

$PYTHON setup.py \
build_ext $BUILD_EXT_FLAGS \
install --single-version-externally-managed \
--record=record.txt
Comment on lines +38 to +41
Copy link
Member

@jakirkham jakirkham May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we able to use pip here?

Copy link
Member

@xhochy xhochy May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check that but in a separate PR.

Copy link
Member

@jakirkham jakirkham May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that makes sense. Was having a little trouble seeing how this was built before. If this is no different from what we were already doing, agree we don't need to change it here.

Copy link
Contributor Author

@pearu pearu May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using pip does not work out of the box:

$PYTHON -m pip install . --no-deps -vv build_ext $BUILD_EXT_FLAGS install

fails with

pip._internal.exceptions.DistributionNotFound: No matching distribution found for build_ext

Copy link
Member

@isuruf isuruf May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need something like,

$PYTHON -m pip install . --no-deps -vv --global-option=build_ext --global-option=--with-cuda

Copy link
Contributor Author

@pearu pearu May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. The following diff works locally:

diff --git a/recipe/build-pyarrow.sh b/recipe/build-pyarrow.sh
index 99396d8..4b01871 100644
--- a/recipe/build-pyarrow.sh
+++ b/recipe/build-pyarrow.sh
@@ -22,7 +22,7 @@ BUILD_EXT_FLAGS=""
 if [[ ! -z "${cuda_compiler_version+x}" && "${cuda_compiler_version}" != "None" ]]
 then
     export PYARROW_WITH_CUDA=1
-    BUILD_EXT_FLAGS="${BUILD_EXT_FLAGS} --with-cuda"
+    BUILD_EXT_FLAGS="${BUILD_EXT_FLAGS} --global-option=build_ext --global-option=--with-cuda"
 else
     export PYARROW_WITH_CUDA=0
 fi
@@ -35,7 +35,4 @@ fi
 
 cd python
 
-$PYTHON setup.py \
-        build_ext $BUILD_EXT_FLAGS \
-        install --single-version-externally-managed \
-                --record=record.txt
+$PYTHON -m pip install . --no-deps -vv $BUILD_EXT_FLAGS
diff --git a/recipe/meta.yaml b/recipe/meta.yaml
index 8a0edf7..20523e2 100644
--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -181,6 +181,7 @@ outputs:
         - boost-cpp
         - cython
         - numpy 1.16.*
+        - pip
         - python
         - setuptools
         - setuptools_scm

Just FYI.

80 changes: 73 additions & 7 deletions recipe/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
{% set version = "0.17.1" %}
{% set number = "2" %}
{% set number = "3" %}
{% set cuda_enabled = cuda_compiler_version is not undefined and cuda_compiler_version == '9.2' %}
{% set build_ext_version = "1.0.0" %}
{% set build_ext = "cuda" if cuda_enabled else "cpu" %}
{% set build_string = "py{}h{}_{}_{}".format(CONDA_PY, PKG_HASH, number, build_ext) %}

package:
name: arrow-cpp-ext
Expand Down Expand Up @@ -40,7 +41,7 @@ outputs:
version: {{ version }}
build:
number: {{ number }}
string: "py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ number }}_{{ build_ext }}"
string: "{{ build_string }}"
skip: true # [cuda_compiler_version not in (undefined, "None", "9.2")]
run_exports:
- {{ pin_subpackage("arrow-cpp", max_pin="x.x.x") }}
Expand All @@ -56,7 +57,7 @@ outputs:
- make # [unix]
- {{ compiler('c') }}
- {{ compiler('cxx') }}
- {{ compiler("cuda") }} # [cuda_compiler_version is not undefined and cuda_compiler_version == '9.2']
- {{ compiler("cuda") }} # [cuda_compiler_version not in (undefined, "None")]
host:
- aws-sdk-cpp
- boost-cpp
Expand All @@ -79,7 +80,6 @@ outputs:
- thrift-cpp >=0.11
- zlib
- zstd

run:
- {{ pin_compatible('numpy', lower_bound='1.16') }}
- aws-sdk-cpp
Expand All @@ -97,7 +97,14 @@ outputs:
- zstd
run_constrained:
- arrow-cpp-proc * {{ build_ext }}
- cudatoolkit >=9.2
- cudatoolkit >=9.2 # [cuda_compiler_version not in (undefined, "None")]

about:
home: https://github.com/apache/arrow
license: Apache-2.0
license_file:
- LICENSE.txt
summary: C++ libraries for Apache Arrow

test:
commands:
Expand Down Expand Up @@ -150,19 +157,78 @@ outputs:
- if exist %PREFIX%\\Library\\lib\\parquet_static.lib exit 1 # [win]
- if exist %PREFIX%\\Library\\lib\\gandiva_static.lib exit 1 # [win]

- name: pyarrow
script: build-pyarrow.sh # [not win]
script: bld-pyarrow.bat # [win]
version: {{ version }}
build:
number: {{ number }}
string: "{{ build_string }}"
skip: true # [cuda_compiler_version not in (undefined, "None", "9.2")]
track_features:
{{ "- arrow-cuda" if cuda_enabled else "" }}
requirements:
build:
- cmake 3.16.*
- ninja
- make # [unix]
- {{ compiler('c') }}
- {{ compiler('cxx') }}
# pyarrow does not require nvcc but it needs to link against libraries in arrow-cpp=*=*cuda
- {{ compiler("cuda") }} # [cuda_compiler_version not in (undefined, "None")]
host:
- {{ pin_subpackage('arrow-cpp', exact=True) }}
- boost-cpp
- cython
- numpy 1.16.*
- python
- setuptools
- setuptools_scm
- six
run:
- {{ pin_subpackage('arrow-cpp', exact=True) }}
- {{ pin_compatible('numpy', lower_bound='1.16') }}
- boost-cpp
- glog
- pandas
# empty parquet-cpp metapackage, force old versions to be uninstalled
- parquet-cpp 1.5.1.*
- python
- six
- snappy
run_constrained:
- arrow-cpp-proc * {{ build_ext }}
- cudatoolkit >=9.2 # [cuda_compiler_version not in (undefined, "None")]

about:
home: https://github.com/apache/arrow
license: Apache-2.0
license_file:
- LICENSE.txt
summary: C++ libraries for Apache Arrow
summary: Python libraries for Apache Arrow

test:
imports:
- pyarrow
- pyarrow.dataset
- pyarrow.flight
- pyarrow.gandiva
- pyarrow.orc # [unix]
- pyarrow.parquet
- pyarrow.plasma # [unix]
- pyarrow.fs
- pyarrow._s3fs
- pyarrow._hdfs
# We can only test importing cuda package but cannot run when a
# CUDA device is not available, for instance, when building from CI.
- pyarrow.cuda # [cuda_compiler_version not in (undefined, "None")]

about:
home: https://github.com/apache/arrow
license: Apache-2.0
license_file:
- LICENSE.txt
summary: C++ libraries for Apache Arrow
summary: C++ and Python libraries for Apache Arrow

extra:
recipe-maintainers:
Expand Down