Skip to content

PyInstaller: エンジン使用時に一時ファイルが残ってしまう問題の修正 #503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Hiroshiba merged 3 commits into from
Nov 3, 2022
Merged

PyInstaller: エンジン使用時に一時ファイルが残ってしまう問題の修正 #503

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ec0ded5
remove onefile option from spec file
y-chan Nov 3, 2022
49a652a
fix dist directory name
y-chan Nov 3, 2022
46753e0
remove unnecessary copy(pysoundfile)
y-chan Nov 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 20 additions & 23 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,28 +429,25 @@ jobs:

if [ -f "download/onnxruntime/lib/onnxruntime_providers_cuda.dll" ]; then
# ONNX Runtime providers (PyInstaller does not copy dynamic loaded libraries)
ln -sf "$(pwd)/download/onnxruntime/lib"/onnxruntime_*.dll dist/run.dist/
ln -sf "$(pwd)/download/onnxruntime/lib"/onnxruntime_*.dll dist/run/

# CUDA
ln -sf "$(pwd)/download/cuda/bin"/cublas64_*.dll dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/cublasLt64_*.dll dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/cudart64_*.dll dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/cufft64_*.dll dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/curand64_*.dll dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/cublas64_*.dll dist/run/
ln -sf "$(pwd)/download/cuda/bin"/cublasLt64_*.dll dist/run/
ln -sf "$(pwd)/download/cuda/bin"/cudart64_*.dll dist/run/
ln -sf "$(pwd)/download/cuda/bin"/cufft64_*.dll dist/run/
ln -sf "$(pwd)/download/cuda/bin"/curand64_*.dll dist/run/

# cuDNN
ln -sf "$(pwd)/download/cudnn/bin"/cudnn64_*.dll dist/run.dist/
ln -sf "$(pwd)/download/cudnn/bin"/cudnn_*_infer64*.dll dist/run.dist/
ln -sf "$(pwd)/download/cudnn/bin"/cudnn64_*.dll dist/run/
ln -sf "$(pwd)/download/cudnn/bin"/cudnn_*_infer64*.dll dist/run/
fi

if [[ ${{ matrix.artifact_name }} == *-directml ]]; then
# DirectML
ln -sf "$(pwd)/download/directml"/DirectML.dll dist/run.dist/
ln -sf "$(pwd)/download/directml"/DirectML.dll dist/run/
fi

# pysoundfile
ln -sf "${{ env.PYTHON_SITE_PACKAGES_DIR }}/_soundfile_data" dist/run.dist/

Comment on lines -451 to -453
Copy link
Copy Markdown
Member Author

@y-chan y-chan Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onefileオプションを取り払ったことで、pyinstallerが勝手にpysoundfileのデータを移行してくれるので、不要になりました(そもそもpyinstallerに移行した時点でこのコピーは不要だった可能性がありますが...)

Copy link
Copy Markdown
Member

@Hiroshiba Hiroshiba Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

調整が必要な場所の調査と反映、ありがとうございます！！

- name: Create symlink of CUDA dependencies
if: startsWith(matrix.os, 'ubuntu-') && endsWith(matrix.artifact_name, 'nvidia')
shell: bash
Expand All @@ -459,29 +456,29 @@ jobs:

# ONNX Runtime providers (PyInstaller does not copy dynamic loaded libraries)
patchelf --set-rpath '$ORIGIN' "$(pwd)/download/onnxruntime/lib"/libonnxruntime_providers_*.so
ln -sf "$(pwd)/download/onnxruntime/lib"/libonnxruntime_*.so dist/run.dist/
ln -sf "$(pwd)/download/onnxruntime/lib"/libonnxruntime_*.so dist/run/

# CUDA
ln -sf "$(pwd)/download/cuda/bin"/libcublas.so.* dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/libcublasLt.so.* dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/libcudart.so.* dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/libcufft.so.* dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/libcurand.so.* dist/run.dist/
ln -sf "$(pwd)/download/cuda/bin"/libcublas.so.* dist/run/
ln -sf "$(pwd)/download/cuda/bin"/libcublasLt.so.* dist/run/
ln -sf "$(pwd)/download/cuda/bin"/libcudart.so.* dist/run/
ln -sf "$(pwd)/download/cuda/bin"/libcufft.so.* dist/run/
ln -sf "$(pwd)/download/cuda/bin"/libcurand.so.* dist/run/

# cuDNN
ln -sf "$(pwd)/download/cudnn/bin"/libcudnn.so.* dist/run.dist/
ln -sf "$(pwd)/download/cudnn/bin"/libcudnn_*_infer.so.* dist/run.dist/
ln -sf "$(pwd)/download/cudnn/bin"/libcudnn.so.* dist/run/
ln -sf "$(pwd)/download/cudnn/bin"/libcudnn_*_infer.so.* dist/run/

- name: Set @rpath to @executable_path
if: startsWith(matrix.os, 'macos-')
run: |
install_name_tool -add_rpath @executable_path/. dist/run.dist/run
install_name_tool -add_rpath @executable_path/. dist/run/run

- name: Code signing
if: github.event.inputs.code_signing == 'true' && startsWith(matrix.os, 'windows-')
shell: bash
run: |
bash build_util/codesign.bash "dist/run.dist/run.exe"
bash build_util/codesign.bash "dist/run/run.exe"
env:
CERT_BASE64: ${{ secrets.CERT_BASE64 }}
CERT_PASSWORD: ${{ secrets.CERT_PASSWORD }}
Expand All @@ -499,7 +496,7 @@ jobs:
# ${{ format('{0}-{1}', matrix.artifact_name, (env.VOICEVOX_ENGINE_VERSION != 'latest' && env.VOICEVOX_ENGINE_VERSION) || github.sha) }}
with:
name: ${{ matrix.artifact_name }}
path: dist/run.dist/
path: dist/run/

upload-to-release:
if: (github.event.release.tag_name || github.event.inputs.version) != ''
Expand Down
67 changes: 30 additions & 37 deletions run.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,31 @@
from PyInstaller.utils.hooks import collect_data_files
import os

datas = collect_data_files('pyopenjtalk')
datas = [
('engine_manifest_assets', 'engine_manifest_assets'),
('speaker_info', 'speaker_info'),
('default.csv', '.'),
('licenses.json', '.'),
('presets.yaml', '.'),
]
datas += collect_data_files('pyopenjtalk')

# コアとONNX Runtimeはバイナリであるが、`binaries`に加えると
# 依存関係のパスがPyInstallerに書き換えらるので、`datas`に加える
# 参考: https://github.com/VOICEVOX/voicevox_engine/pull/446#issuecomment-1210052318
libcore_path = os.environ.get('LIBCORE_PATH')
if libcore_path:
print('LIBCORE_PATH is found:', libcore_path)
if not os.path.isfile(libcore_path):
raise Exception("LIBCORE_PATH was found, but it is not file!")
datas += [(libcore_path, ".")]

libonnxruntime_path = os.environ.get('LIBONNXRUNTIME_PATH')
if libonnxruntime_path:
print('LIBONNXRUNTIME_PATH is found:', libonnxruntime_path)
if not os.path.isfile(libonnxruntime_path):
raise Exception("LIBCORE_PATH was found, but it is not file!")
datas += [(libonnxruntime_path, ".")]


block_cipher = None
Expand All @@ -30,17 +54,13 @@ pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
exclude_binaries=True,
name='run',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
upx_exclude=[],
runtime_tmpdir=None,
console=True,
disable_windowed_traceback=False,
argv_emulation=False,
Expand All @@ -49,40 +69,13 @@ exe = EXE(
entitlements_file=None,
)

datas = [
('engine_manifest_assets', 'engine_manifest_assets', 'DATA'),
('speaker_info', 'speaker_info', 'DATA'),
('default.csv', 'default.csv', 'DATA'),
('licenses.json', 'licenses.json', 'DATA'),
('presets.yaml', 'presets.yaml', 'DATA'),
]

# コアとONNX Runtimeはバイナリであるが、`binaries`に加えると
# 依存関係のパスがPyInstallerに書き換えらるので、`datas`に加える
# 参考: https://github.com/VOICEVOX/voicevox_engine/pull/446#issuecomment-1210052318
libcore_path = os.environ.get('LIBCORE_PATH')
if libcore_path:
print('LIBCORE_PATH is found:', libcore_path)
if not os.path.isfile(libcore_path):
raise Exception("LIBCORE_PATH was found, but it is not file!")
filename = os.path.basename(libcore_path)
datas += [(filename, libcore_path, 'DATA')]

libonnxruntime_path = os.environ.get('LIBONNXRUNTIME_PATH')
if libonnxruntime_path:
print('LIBONNXRUNTIME_PATH is found:', libonnxruntime_path)
if not os.path.isfile(libonnxruntime_path):
raise Exception("LIBCORE_PATH was found, but it is not file!")
filename = os.path.basename(libonnxruntime_path)
datas += [(filename, libonnxruntime_path, 'DATA')]

coll = COLLECT(
exe,
[],
[],
datas,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='run.dist',
name='run',
Comment on lines -87 to +80
Copy link
Copy Markdown
Member Author

@y-chan y-chan Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onefileを使用して、かつ必要なフォルダ(speaker_info/engine_manifest_assets)やファイル(default.csv/onnxruntime.dllなど)のコピー処理も行う場合、成果物が生成されるdistフォルダ内にrunという実行ファイルとrunというフォルダが生成されることになり、この場合ファイル操作が壊れることがあるので、出力フォルダ名をrun.distと変更するためにこのような変更をしていました。
onefileを使用しなくなったことで、distフォルダ内にはrunフォルダのみが生成されるようになったので、名前を戻しました。(CIの変更もこれによるものです)

)