Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add torchaudio #27715

Merged
merged 70 commits into from
Oct 13, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
264666a
add torchaudio
hadim Sep 28, 2024
5c06f57
ninja
hadim Sep 28, 2024
4c391cf
remove fairseq
hadim Sep 28, 2024
3078b4e
ci env
hadim Sep 28, 2024
67e56b5
false/true
hadim Sep 28, 2024
43be3b7
no tests
hadim Sep 28, 2024
ffe289f
Merge branch 'main' into torchaudio
hadim Sep 29, 2024
ef0c734
Update recipe.yaml
hadim Sep 30, 2024
a8cd229
Merge branch 'main' into torchaudio
hadim Sep 30, 2024
d054089
Update recipe.yaml
hadim Sep 30, 2024
db0183c
Merge branch 'main' into torchaudio
hadim Sep 30, 2024
df8ad65
Merge branch 'main' into torchaudio
hadim Oct 1, 2024
bbc4c2f
Merge branch 'main' into torchaudio
hadim Oct 1, 2024
1becc34
run tests
hadim Oct 2, 2024
66b0c26
Merge branch 'main' into torchaudio
hadim Oct 2, 2024
013a142
Merge branch 'main' into torchaudio
hadim Oct 4, 2024
722a832
wip
hadim Oct 4, 2024
d6f462e
wip
hadim Oct 4, 2024
3bcbce9
WIP
hadim Oct 4, 2024
0f0739c
WIP
hadim Oct 4, 2024
3618216
Disable ROCm for now; fix FFMPEG_ROOT
Tobias-Fischer Oct 6, 2024
87859da
Add missing test deps; add missing third party license
Tobias-Fischer Oct 6, 2024
f2dfc8d
Add missing skip for test
Tobias-Fischer Oct 6, 2024
710cb99
Skip some failing tests
Tobias-Fischer Oct 6, 2024
110ab56
Lint
Tobias-Fischer Oct 6, 2024
d0c3aaf
Disable ffmpeg/sox/rnnt
Tobias-Fischer Oct 7, 2024
3f91b04
Skip sox/ffmpeg tests
Tobias-Fischer Oct 7, 2024
6ce9f55
Skip more tests
Tobias-Fischer Oct 8, 2024
134dbfc
Try fix cuda issues
Tobias-Fischer Oct 8, 2024
172843b
Skip segfault test
Tobias-Fischer Oct 8, 2024
8f073f9
Do not install sox/ffmpeg
Tobias-Fischer Oct 8, 2024
07569a2
Use workaround from https://github.com/conda-forge/staged-recipes/pul…
Tobias-Fischer Oct 8, 2024
3778715
Update run_tests
Tobias-Fischer Oct 8, 2024
826dd50
Skip
Tobias-Fischer Oct 8, 2024
5988179
Next round of addressing segfaults
Tobias-Fischer Oct 9, 2024
5b7d0f6
Swap maintainers
Tobias-Fischer Oct 9, 2024
ac7d7bc
Tidy up run_tests.sh
Tobias-Fischer Oct 9, 2024
1d21508
One more failing test
Tobias-Fischer Oct 9, 2024
4098634
Skip more segfault tests
Tobias-Fischer Oct 9, 2024
ac4b49d
Debug nvcc for cuda12
Tobias-Fischer Oct 9, 2024
d802a77
Typo
Tobias-Fischer Oct 9, 2024
7999fea
More segfaults
Tobias-Fischer Oct 9, 2024
85793f0
Use CUDACXX
Tobias-Fischer Oct 9, 2024
2154c08
Segfault
Tobias-Fischer Oct 9, 2024
ae73876
Segfault
Tobias-Fischer Oct 9, 2024
1763fd5
Only build cuda 12
Tobias-Fischer Oct 9, 2024
8b701fd
Devug
Tobias-Fischer Oct 9, 2024
c39e7a0
Update build.sh
Tobias-Fischer Oct 9, 2024
befe6b0
Create 0002-use-conda-cuda.patch
Tobias-Fischer Oct 9, 2024
869700b
use 0002-use-conda-cuda.patch
Tobias-Fischer Oct 9, 2024
4d01e36
Fixup
Tobias-Fischer Oct 9, 2024
64fdf3a
Formatting
Tobias-Fischer Oct 9, 2024
dbe3eb9
Update recipe.yaml
Tobias-Fischer Oct 9, 2024
12ba915
Update build.sh
Tobias-Fischer Oct 9, 2024
d361e0b
Update build.sh
Tobias-Fischer Oct 9, 2024
5516bb6
explicitly set CUDA_INCLUDE_DIRS
Tobias-Fischer Oct 9, 2024
d7c5ef6
Create 0003-Apply-CMAKE_ARGS-if-set.patch
Tobias-Fischer Oct 9, 2024
777196b
Update recipe.yaml
Tobias-Fischer Oct 9, 2024
69a0116
Update build.sh
Tobias-Fischer Oct 9, 2024
e62b41f
Debug
Tobias-Fischer Oct 10, 2024
796a15c
Update build.sh
Tobias-Fischer Oct 10, 2024
459d54e
Update build.sh
Tobias-Fischer Oct 10, 2024
c01e56a
Fixup
Tobias-Fischer Oct 10, 2024
137bcbc
Cleanup
Tobias-Fischer Oct 10, 2024
894a89f
Enable all builds again to see if they still work
Tobias-Fischer Oct 10, 2024
240eab8
Add all builds, add @h-vetinari as maintainer
Tobias-Fischer Oct 10, 2024
f3ae343
Rename 3837.patch to 0004-add-missing-ffmpeg-skip.patch
Tobias-Fischer Oct 10, 2024
4162e17
Update recipe.yaml
Tobias-Fischer Oct 10, 2024
415e262
Merge branch 'main' into torchaudio
Tobias-Fischer Oct 10, 2024
c17800d
A few more skips
Tobias-Fischer Oct 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions recipes/torchaudio/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/bin/bash
set -ex

if [[ ${cuda_compiler_version} != "None" ]]; then
# Set the CUDA arch list from
# https://github.com/conda-forge/pytorch-cpu-feedstock/blob/main/recipe/build_pytorch.sh

if [[ ${cuda_compiler_version} == 11.8 ]]; then
export TORCH_CUDA_ARCH_LIST="3.5;5.0;6.0;6.1;7.0;7.5;8.0;8.6;8.9+PTX"
export CUDA_TOOLKIT_ROOT_DIR=$CUDA_HOME
elif [[ ${cuda_compiler_version} == 12.0 ]]; then
export TORCH_CUDA_ARCH_LIST="5.0;6.0;6.1;7.0;7.5;8.0;8.6;8.9;9.0+PTX"
# $CUDA_HOME not set in CUDA 12.0. Using $PREFIX
export CUDA_TOOLKIT_ROOT_DIR="${PREFIX}"
else
echo "unsupported cuda version. edit build.sh"
exit 1
fi

export USE_ROCM="true"
export USE_CUDA="true"
export BUILD_CUDA_CTC_DECODER="true"

else
export USE_ROCM="false"
export USE_CUDA="false"
export BUILD_CUDA_CTC_DECODER="false"
fi

export BUILD_CPP_TEST="false"
export BUILD_SOX="true"
export BUILD_RIR="true"
export BUILD_RNNT="true"
export BUILD_ALIGN="true"
export USE_FFMPEG="true"

This comment was marked as resolved.

This comment was marked as resolved.

export USE_OPENMP="true"

export CMAKE_C_COMPILER="$CC"
export CMAKE_CXX_COMPILER="$CXX"
export CMAKE_GENERATOR="Ninja"

python -m pip install . -vv
16 changes: 16 additions & 0 deletions recipes/torchaudio/patches/0001-point-to-correct-prefix.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
diff --git a/tools/setup_helpers/extension.py b/tools/setup_helpers/extension.py
index 2415bbae..f724c4b8 100644
--- a/tools/setup_helpers/extension.py
+++ b/tools/setup_helpers/extension.py
@@ -124,8 +124,9 @@ class CMakeBuild(build_ext):

cmake_args = [
f"-DCMAKE_BUILD_TYPE={cfg}",
- f"-DCMAKE_PREFIX_PATH={torch.utils.cmake_prefix_path}",
- f"-DCMAKE_INSTALL_PREFIX={extdir}",
+ f"-DCMAKE_PREFIX_PATH={os.environ['PREFIX']}",
+ f"-DCMAKE_LIBRARY_PREFIX={os.environ['PREFIX']}",
+ f"-DCMAKE_INSTALL_PREFIX={os.environ['SP_DIR']}/torch/lib",
"-DCMAKE_VERBOSE_MAKEFILE=ON",
f"-DPython_INCLUDE_DIR={distutils.sysconfig.get_python_inc()}",
f"-DBUILD_CPP_TEST={'ON' if _BUILD_CPP_TEST else 'OFF'}",
167 changes: 167 additions & 0 deletions recipes/torchaudio/recipe.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
context:
name: torchaudio
version: "2.4.1"
build_number: 0
cuda_build_string: cuda_${{ cuda_version | version_to_buildstring }}
string_prefix: ${{ cuda_build_string if cuda else "cpu_" }}

package:
name: ${{ name|lower }}
version: ${{ version }}

source:
- url: https://github.com/pytorch/audio/archive/refs/tags/v${{ version }}.tar.gz
sha256: a6b7e1b606ae353b9c4db1c8e754263f53f0457e9ab538e9669c0af0c46a0d5c
patches:
- patches/0001-point-to-correct-prefix.patch

build:
number: ${{ build_number}}
skip:
- win
string: ${{ string_prefix }}py${{ python | version_to_buildstring }}h${{ hash }}_${{ build_number }}
variant:
use_keys:
# use cuda from the variant config, e.g. to build multiple CUDA variants
- cuda
# this will down-prioritize the cuda variant versus other variants of the package
down_prioritize_variant: ${{ 1 if cuda else 0 }}
hadim marked this conversation as resolved.
Show resolved Hide resolved
dynamic_linking:
rpaths:
- lib/
hadim marked this conversation as resolved.
Show resolved Hide resolved
script:
file: build
env:
cuda_compiler_version: ${{ cuda_version | default('None') }}

requirements:
build:
- if: build_platform != target_platform
then:
- python
- cross-python_{{ target_platform }}

- ${{ compiler('cxx') }}
- ${{ stdlib("c") }}
- cmake
- ninja

- if: cuda
then:
- ${{ compiler('cuda') }}
- cuda-version ==${{ cuda_version }}

- if: cuda and match(cuda_version, ">=12") and build_platform != target_platform
then:
- cuda-driver-dev
- cuda-cudart-dev
- cuda-nvrtc-dev
- cuda-nvtx-dev
- cuda-nvml-dev
- libcublas-dev
- libcufft-dev
- libcurand-dev
- libcusolver-dev
- libcusparse-dev

host:
- python
- pip
- setuptools
- pytorch
- ${{ "pytorch =*=cuda*" if cuda == "yes" }}
- ${{ "pytorch =*=cpu*" if cuda == "no" }}

- bzip2
- kaldi
- pybind11
- sox
- xz
- zlib

- ${{ "cuda-version =={{ cuda_version }}" if cuda }}

- if: cuda and match(cuda_version, ">=12")
then:
- cuda-driver-dev
- cuda-cudart-dev
- cuda-nvrtc-dev
- cuda-nvtx-dev
- cuda-nvml-dev
- libcublas-dev
- libcufft-dev
- libcurand-dev
- libcusolver-dev
- libcusparse-dev
run:
- python
- numpy
- kaldi
- ${{ "pytorch =*=cuda*" if cuda == "yes" }}
- ${{ "pytorch =*=cpu*" if cuda == "no" }}

ignore_run_exports:
from_package:
- if: cuda and match(cuda_version, ">=12")
then:
- cuda-nvrtc-dev
- cuda-nvtx-dev
- libcublas-dev
- libcufft-dev
- libcurand-dev
- libcusolver-dev
- libcusparse-dev

tests:
- python:
imports:
- torchaudio
- torchaudio.backend
- torchaudio.compliance
- torchaudio.datasets
- torchaudio.functional
- torchaudio.models
- torchaudio.pipelines
- torchaudio.kaldi_io
- torchaudio.utils
- torchaudio.sox_effects
- torchaudio.transforms
pip_check: true

- requirements:
run:
- pytest
- scipy
- numpy
- librosa
- expecttest
- requests
- hypothesis
- inflect
- kaldi_io
- parameterized
- pysoundfile
- transformers
- unidecode
- inflect
files:
source:
- test/
- examples/
script:
# remove example tests that are not importable out-of-tree
# - rm -rf test/torchaudio_unittest/example
# - CI=true pytest -v test/

# Note: Tests take too long to run and some are not importable out-of-tree
- echo "Hello World"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see some result of running the test suite (and efforts to get it to run) before we decide to skip it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I enabled them, let's see what happens.


about:
homepage: https://github.com/pytorch/audio
license: BSD-2-Clause
license_file: LICENSE
summary: Data manipulation and transformation for audio signal processing, powered by PyTorch

extra:
recipe-maintainers:
- hadim
Loading