Skip to content

WindowsビルドをPyInstallerへ移行 #443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
y-chan merged 1 commit into from
Aug 7, 2022
Merged

WindowsビルドをPyInstallerへ移行 #443

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 14 additions & 71 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -379,28 +379,22 @@ jobs:
architecture: "x64"
voicevox_core_asset_prefix: voicevox_core-windows-x64-cpu
onnxruntime_url: https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/onnxruntime-win-x64-1.10.0.zip
ccache_url: https://github.com/ccache/ccache/releases/download/v4.4.1/ccache-4.4.1-windows-64.zip
artifact_name: windows-cpu
nuitka_cache_path: nuitka_cache
# Windows DirectML
- os: windows-2019
architecture: "x64"
voicevox_core_asset_prefix: voicevox_core-windows-x64-directml
onnxruntime_url: https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/Microsoft.ML.OnnxRuntime.DirectML.1.10.0.zip
directml_url: https://www.nuget.org/api/v2/package/Microsoft.AI.DirectML/1.8.0
ccache_url: https://github.com/ccache/ccache/releases/download/v4.4.1/ccache-4.4.1-windows-64.zip
artifact_name: windows-directml
nuitka_cache_path: nuitka_cache
# Windows NVIDIA GPU
- os: windows-2019
architecture: "x64"
voicevox_core_asset_prefix: voicevox_core-windows-x64-cuda
onnxruntime_url: https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/onnxruntime-win-x64-gpu-1.10.0.zip
cuda_version: "11.4.2"
cudnn_url: https://developer.download.nvidia.com/compute/redist/cudnn/v8.2.4/cudnn-11.4-windows-x64-v8.2.4.15.zip
ccache_url: https://github.com/ccache/ccache/releases/download/v4.4.1/ccache-4.4.1-windows-64.zip
artifact_name: windows-nvidia
nuitka_cache_path: nuitka_cache

runs-on: ${{ matrix.os }}

Expand Down Expand Up @@ -501,9 +495,6 @@ jobs:
- name: Install Python dependencies
shell: bash
run: |
# FIXME: Nuitka cannot build with setuptools>=60.7.0
# https://github.com/Nuitka/Nuitka/issues/1406
python -m pip install --upgrade pip setuptools==60.6.0 wheel
python -m pip install -r requirements-dev.txt

# Download pyopenjtalk dictionary
Expand All @@ -527,32 +518,6 @@ jobs:
shell: bash
run: mkdir -p download/

# Install Ccache
- name: Export Ccache url to calc hash
shell: bash
run: echo "${{ matrix.ccache_url }}" > download/ccache_url.txt

- name: Prepare Ccache
uses: actions/cache@v2
id: ccache-cache
with:
key: ${{ matrix.os }}-ccache-${{ hashFiles('download/ccache_url.txt') }}
path: download/ccache

- name: Download Ccache
if: steps.ccache-cache.outputs.cache-hit != 'true'
shell: bash
run: |
curl -L "${{ matrix.ccache_url }}" > download/ccache.zip
unzip download/ccache.zip -d download/
rm download/ccache.zip
mv download/ccache-*/ download/ccache

- name: Install Ccache
shell: bash
run: |
echo "$HOME/download/ccache" >> $GITHUB_PATH

# Donwload DirectML
- name: Export DirectML url to calc hash
if: endswith(matrix.artifact_name, '-directml')
Expand Down Expand Up @@ -672,52 +637,30 @@ jobs:
# FIXME: VOICEVOX (editor) cannot build without licenses.json
cp engine_manifest_assets/dependency_licenses.json licenses.json

- name: Cache Nuitka (ccache, module-cache)
uses: actions/cache@v2
id: nuitka-cache
with:
path: ${{ matrix.nuitka_cache_path }}
key: ${{ runner.os }}-nuitka-${{ matrix.artifact_name }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-nuitka-${{ matrix.artifact_name }}-

- name: Show disk space (debug info)
shell: bash
run: |
df -h

- name: Build run.py with Nuitka
- name: Build run.py with PyInstaller
shell: bash
env:
NUITKA_CACHE_DIR: ${{ matrix.nuitka_cache_path }}
run: |
set -eux

# Replace version
sed -i "s/__version__ = \"latest\"/__version__ = \"${{ env.VOICEVOX_ENGINE_VERSION }}\"/" voicevox_engine/__init__.py

python -m nuitka \
--standalone \
--assume-yes-for-downloads \
--plugin-enable=numpy \
--plugin-enable=multiprocessing \
--follow-import-to=numpy \
--follow-import-to=aiofiles \
--include-package=uvicorn \
--include-package=anyio \
--include-package-data=pyopenjtalk \
--include-package-data=scipy \
--include-data-file="licenses.json=./" \
--include-data-file="presets.yaml=./" \
--include-data-file="default.csv=./" \
--include-data-file="engine_manifest.json=./" \
--include-data-file="download/onnxruntime/lib/onnxruntime.dll=./" \
--include-data-file="download/core/core.dll=./" \
--include-data-dir="speaker_info=./speaker_info" \
--include-data-dir="engine_manifest_assets=./engine_manifest_assets" \
--msvc=14.2 \
--follow-imports \
--no-prefer-source-code \
pyinstaller \
--noconfirm \
--collect-data pyopenjtalk \
--add-data "licenses.json;." \
--add-data "presets.yaml;." \
--add-data "default.csv;." \
--add-data "engine_manifest.json;." \
--add-data "speaker_info;speaker_info" \
--add-data "engine_manifest_assets;engine_manifest_assets" \
--add-binary "download/onnxruntime/lib/onnxruntime.dll;." \
--add-binary "download/core/core.dll;." \
Comment on lines +653 to +663
Copy link
Copy Markdown
Member

@PickledChair PickledChair Aug 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

（pyinstaller を使っていたことがあるのですが、だいぶ以前なので記憶が錆び付いています……。間違った情報を提供していたらすみません）

こちらのオプションは spec ファイルにまとめると、CI だけでなく手元でビルドする時も便利だと思いました（spec ファイルについては https://pyinstaller.org/en/stable/spec-files.html を参照）。spec ファイルは実際には Pythonのソースファイルなので、OS ごとに設定を記述するということもできる気がします。また、もしかしたら全てのオプションを spec ファイルに書けないかもしれないですが、その場合でも pyinstaller コマンド実行時に追加のオプションを指定できたと思います。

ここのオプションが問題ないかどうかはまだ確認できていないです。何かわかったらまたコメントしたいと思います。

Copy link
Copy Markdown
Member Author

@y-chan y-chan Aug 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

specを記述すると、手元でビルドする際に、各人にspecを書き換えてもらう必要が出るなと思ったので、オプションにしました。
specの方が利便性が高いとは思うのですが、実際どうなんでしょうか...

Copy link
Copy Markdown
Member

@Hiroshiba Hiroshiba Aug 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

複雑な動的処理がある場合はspecファイルのほうが便利かもです。
現状はファイル指定だけなのでどちらでも良いかなと感じました。
どちらかというと、actionsの特性上specファイルとbuild.ymlのファイルパスが非常に離れるので、一箇所に書ける今のほうが見通しが良いかもしれません。

とりあえずwin版だけの今はこの状態にしておいて、mac版やlinux版を作ったり、より複雑な操作が必要になったときに再考するのはどうでしょう。

Copy link
Copy Markdown
Member Author

@y-chan y-chan Aug 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

specファイルを使えば、必要のないディレクトリ・ファイルの削除などが一括で出来るのですが、download/onnxruntime/lib/onnxruntime.dll;などのオプションはOSごとに変わってくるため、OSごとにspecファイルを書かなければならず、面倒であるので今回はパスしました。
まあ、こういった動作は別にcpコマンドに分けるようにしてしまってもいいかもしれません。

一旦はこのままにして、後で考え直すようにしようかなと思います

Copy link
Copy Markdown
Member

@Hiroshiba Hiroshiba Aug 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

specファイル（pythonファイル）の中でOSで判定してロジックを切り替える、という手もあると思います。
･･･が、ダウンロードしたdllのパスの切り替えは、spec内ではなくactions内でやったほうが絶対良さそう。。

一旦このまま、賛成です！

Copy link
Copy Markdown
Member

@PickledChair PickledChair Aug 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spec ファイルの件、了解いたしました！　CI を書く上での利便性やわかりやすさ等は確かに spec ファイルではなく明示的なオプション指定の方がよさそうですね（そこまでオプションの数も多くないので）。いったんこのままということで納得しました……！

run.py

- name: Show disk space (debug info)
Expand All @@ -737,12 +680,12 @@ jobs:

# Build artifact directory
mkdir -p artifact
ln -sf "$(pwd)/run.dist"/* artifact/
ln -sf "$(pwd)/dist/run"/* artifact/

# Copy DLL dependencies

if [ -f "download/onnxruntime/lib/onnxruntime_providers_cuda.dll" ]; then
# ONNX Runtime providers (Nuitka does not copy dynamic loaded libraries)
# ONNX Runtime providers (PyInstaller does not copy dynamic loaded libraries)
ln -sf "$(pwd)/download/onnxruntime/lib"/onnxruntime_*.dll artifact/

# CUDA
Expand Down
9 changes: 7 additions & 2 deletions generate_licenses.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,9 +161,14 @@ def generate_licenses() -> List[License]:
if license.text == "UNKNOWN":
if license.name.lower() == "core" and license.version == "0.0.0":
continue
elif license.name.lower() == "nuitka":
elif license.name.lower() == "future":
with urllib.request.urlopen(
"https://raw.githubusercontent.com/Nuitka/Nuitka/develop/LICENSE.txt"
"https://raw.githubusercontent.com/PythonCharmers/python-future/master/LICENSE.txt" # noqa: B950
) as res:
license.text = res.read().decode()
elif license.name.lower() == "pefile":
with urllib.request.urlopen(
"https://raw.githubusercontent.com/erocarrera/pefile/master/LICENSE" # noqa: B950
) as res:
license.text = res.read().decode()
elif license.name.lower() == "pyopenjtalk":
Expand Down
2 changes: 1 addition & 1 deletion requirements-dev.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
-r requirements.in
cython
nuitka
pyinstaller
pip-licenses
pip-tools
pre-commit
Expand Down
26 changes: 23 additions & 3 deletions requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
#
aiofiles==0.7.0
# via -r requirements.in
altgraph==0.17.2
# via pyinstaller
anyio==3.3.4
# via starlette
appdirs==1.4.4
Expand All @@ -16,10 +18,14 @@ atomicwrites==1.4.0
# via -r requirements-dev.in
backports.entry-points-selectable==1.1.1
# via virtualenv
certifi==2022.6.15
# via requests
cffi==1.15.0
# via soundfile
cfgv==3.3.1
# via pre-commit
charset-normalizer==2.1.0
# via requests
click==8.0.3
# via
# pip-tools
Expand All @@ -40,22 +46,26 @@ fastapi==0.70.0
# via -r requirements.in
filelock==3.4.0
# via virtualenv
future==0.18.2
# via pefile
h11==0.12.0
# via uvicorn
identify==2.4.0
# via pre-commit
idna==3.3
# via anyio
# via
# anyio
# requests
nodeenv==1.6.0
# via pre-commit
nuitka==0.6.17.4
# via -r requirements-dev.in
numpy==1.20.0
# via
# -r requirements.in
# pyopenjtalk
# pyworld
# scipy
pefile==2022.5.30
# via pyinstaller
pep517==0.12.0
# via pip-tools
pip-licenses==3.5.3
Expand All @@ -72,16 +82,24 @@ pycparser==2.20
# via cffi
pydantic==1.8.2
# via fastapi
pyinstaller==5.3
# via -r requirements-dev.in
pyinstaller-hooks-contrib==2022.8
# via pyinstaller
pyopenjtalk @ git+https://github.com/VOICEVOX/pyopenjtalk@50b0296a9e1b666e5a09a41ec9e9284a2a9b608f
# via -r requirements.in
python-multipart==0.0.5
# via -r requirements.in
pywin32-ctypes==0.2.0
# via pyinstaller
pyworld==0.3.0
# via -r requirements.in
pyyaml==6.0
# via
# -r requirements.in
# pre-commit
requests==2.28.1
# via -r requirements.in
scipy==1.7.1
# via -r requirements.in
six==1.16.0
Expand All @@ -103,6 +121,8 @@ tqdm==4.62.3
# via pyopenjtalk
typing-extensions==3.10.0.2
# via pydantic
urllib3==1.26.11
# via requests
uvicorn==0.15.0
# via -r requirements.in
virtualenv==20.10.0
Expand Down