Docker container crashes #162
Description
Hi,
I just wanted to try out the repo using the Docker server (running Docker on Windows). I can start the container using docker compose up, however the container takes quite long to load everything required and then crashes after approx. 8 minutes (without me sending any requests from a client). Also, since the crash seems related to parlerTTS, I tried using Melo (adding --tts melo to the command in the docker-compose file). That also does not work. See related logs below.
- Is the Dockerfile maintained / working?
- Is the Dockerfile even built to run one of the other TTS, or will it only work with parler without further adjustments?
- Any suggestions on how to fix the errors?
Thanks in advance for any reply!
Cheers
When using parlerTTS (the default), here is the log output from start to crash:
Click to expand
2025-04-13 13:16:02 pipeline-1 |
2025-04-13 13:16:02 pipeline-1 | ==========
2025-04-13 13:16:02 pipeline-1 | == CUDA ==
2025-04-13 13:16:02 pipeline-1 | ==========
2025-04-13 13:16:02 pipeline-1 |
2025-04-13 13:16:02 pipeline-1 | CUDA Version 12.1.1
2025-04-13 13:16:02 pipeline-1 |
2025-04-13 13:16:02 pipeline-1 | Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2025-04-13 13:16:02 pipeline-1 |
2025-04-13 13:16:02 pipeline-1 | This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2025-04-13 13:16:02 pipeline-1 | By pulling and using the container, you accept the terms and conditions of this license:
2025-04-13 13:16:02 pipeline-1 | https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2025-04-13 13:16:02 pipeline-1 |
2025-04-13 13:16:02 pipeline-1 | A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2025-04-13 13:16:02 pipeline-1 |
2025-04-13 13:16:04 pipeline-1 | /opt/conda/lib/python3.11/site-packages/df/io.py:9: UserWarning: `torchaudio.backend.common.AudioMetaData` has been moved to `torchaudio.AudioMetaData`. Please update the import path.
2025-04-13 13:16:04 pipeline-1 | from torchaudio.backend.common import AudioMetaData
2025-04-13 13:16:06 pipeline-1 | [nltk_data] Downloading package punkt_tab to /root/nltk_data...
2025-04-13 13:16:08 pipeline-1 | [nltk_data] Unzipping tokenizers/punkt_tab.zip.
2025-04-13 13:16:08 pipeline-1 | [nltk_data] Downloading package averaged_perceptron_tagger_eng to
2025-04-13 13:16:08 pipeline-1 | [nltk_data] /root/nltk_data...
2025-04-13 13:16:08 pipeline-1 | [nltk_data] Unzipping taggers/averaged_perceptron_tagger_eng.zip.
2025-04-13 13:16:09 pipeline-1 | Using cache found in /root/.cache/torch/hub/snakers4_silero-vad_master
2025-04-13 13:16:58 pipeline-1 | 2025-04-13 11:16:58,325 - STT.whisper_stt_handler - INFO - Warming up WhisperSTTHandler
2025-04-13 13:16:58 pipeline-1 | You have passed task=transcribe, but also have set `forced_decoder_ids` to [[1, None], [2, 50360]] which creates a conflict. `forced_decoder_ids` will be ignored in favor of task=transcribe.
2025-04-13 13:16:58 pipeline-1 | The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
2025-04-13 13:17:46 pipeline-1 | 2025-04-13 11:17:46,688 - STT.whisper_stt_handler - INFO - WhisperSTTHandler: warmed up! time: 47.364 s
2025-04-13 13:17:47 pipeline-1 | 2025-04-13 11:17:47,795 - transformers_modules.microsoft.Phi-3-mini-4k-instruct.0a67737cc96d2554230f90338b163bc6380a2a85.modeling_phi3 - WARNING - `flash-attention` package not found, consider installing for better performance: No module named 'flash_attn'.
2025-04-13 13:17:47 pipeline-1 | 2025-04-13 11:17:47,795 - transformers_modules.microsoft.Phi-3-mini-4k-instruct.0a67737cc96d2554230f90338b163bc6380a2a85.modeling_phi3 - WARNING - Current `flash-attention` does not support `window_size`. Either upgrade or use `attn_implementation='eager'`.
2025-04-13 13:21:40 pipeline-1 |
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [02:32<02:32, 152.51s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [03:52<00:00, 109.72s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [03:52<00:00, 116.14s/it]
2025-04-13 13:21:42 pipeline-1 | 2025-04-13 11:21:42,613 - LLM.language_model - INFO - Warming up LanguageModelHandler
2025-04-13 13:21:42 pipeline-1 | /opt/conda/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:590: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
2025-04-13 13:21:42 pipeline-1 | warnings.warn(
2025-04-13 13:21:42 pipeline-1 | The `seen_tokens` attribute is deprecated and will be removed in v4.41. Use the `cache_position` model input instead.
2025-04-13 13:21:42 pipeline-1 | `get_max_cache()` is deprecated for all Cache classes. Use `get_max_cache_shape()` instead. Calling `get_max_cache()` will raise error from v4.48
2025-04-13 13:21:42 pipeline-1 | 2025-04-13 11:21:42,733 - transformers_modules.microsoft.Phi-3-mini-4k-instruct.0a67737cc96d2554230f90338b163bc6380a2a85.modeling_phi3 - WARNING - You are not running the flash-attention implementation, expect numerical differences.
2025-04-13 13:21:43 pipeline-1 | 2025-04-13 11:21:43,047 - LLM.language_model - INFO - LanguageModelHandler: warmed up! time: 0.460 s
2025-04-13 13:21:43 pipeline-1 | 2025-04-13 11:21:43,440 - parler_tts.modeling_parler_tts - WARNING - Flash attention 2 is not installed
2025-04-13 13:21:43 pipeline-1 | 2025-04-13 11:21:43,975 - TTS.parler_handler - WARNING - Parler TTS works best with flash attention 2, but is not installed
2025-04-13 13:21:43 pipeline-1 | Given that CUDA is available in this system, you can install flash attention 2 with `uv pip install flash-attn --no-build-isolation`
2025-04-13 13:22:03 pipeline-1 | /opt/conda/lib/python3.11/site-packages/torch/nn/utils/weight_norm.py:134: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
2025-04-13 13:22:03 pipeline-1 | WeightNorm.apply(module, name, dim)
2025-04-13 13:22:25 pipeline-1 | 2025-04-13 11:22:25,615 - parler_tts.modeling_parler_tts - WARNING - Config of the text_encoder: <class 'transformers.models.t5.modeling_t5.T5EncoderModel'> is overwritten by shared text_encoder config: T5Config {
2025-04-13 13:22:25 pipeline-1 | "_name_or_path": "google/flan-t5-large",
2025-04-13 13:22:25 pipeline-1 | "architectures": [
2025-04-13 13:22:25 pipeline-1 | "T5ForConditionalGeneration"
2025-04-13 13:22:25 pipeline-1 | ],
2025-04-13 13:22:25 pipeline-1 | "classifier_dropout": 0.0,
2025-04-13 13:22:25 pipeline-1 | "d_ff": 2816,
2025-04-13 13:22:25 pipeline-1 | "d_kv": 64,
2025-04-13 13:22:25 pipeline-1 | "d_model": 1024,
2025-04-13 13:22:25 pipeline-1 | "decoder_start_token_id": 0,
2025-04-13 13:22:25 pipeline-1 | "dense_act_fn": "gelu_new",
2025-04-13 13:22:25 pipeline-1 | "dropout_rate": 0.1,
2025-04-13 13:22:25 pipeline-1 | "eos_token_id": 1,
2025-04-13 13:22:25 pipeline-1 | "feed_forward_proj": "gated-gelu",
2025-04-13 13:22:25 pipeline-1 | "initializer_factor": 1.0,
2025-04-13 13:22:25 pipeline-1 | "is_encoder_decoder": true,
2025-04-13 13:22:25 pipeline-1 | "is_gated_act": true,
2025-04-13 13:22:25 pipeline-1 | "layer_norm_epsilon": 1e-06,
2025-04-13 13:22:25 pipeline-1 | "model_type": "t5",
2025-04-13 13:22:25 pipeline-1 | "n_positions": 512,
2025-04-13 13:22:25 pipeline-1 | "num_decoder_layers": 24,
2025-04-13 13:22:25 pipeline-1 | "num_heads": 16,
2025-04-13 13:22:25 pipeline-1 | "num_layers": 24,
2025-04-13 13:22:25 pipeline-1 | "output_past": true,
2025-04-13 13:22:25 pipeline-1 | "pad_token_id": 0,
2025-04-13 13:22:25 pipeline-1 | "relative_attention_max_distance": 128,
2025-04-13 13:22:25 pipeline-1 | "relative_attention_num_buckets": 32,
2025-04-13 13:22:25 pipeline-1 | "tie_word_embeddings": false,
2025-04-13 13:22:25 pipeline-1 | "transformers_version": "4.46.1",
2025-04-13 13:22:25 pipeline-1 | "use_cache": true,
2025-04-13 13:22:25 pipeline-1 | "vocab_size": 32128
2025-04-13 13:22:25 pipeline-1 | }
2025-04-13 13:22:25 pipeline-1 |
2025-04-13 13:22:25 pipeline-1 | 2025-04-13 11:22:25,615 - parler_tts.modeling_parler_tts - WARNING - Config of the audio_encoder: <class 'parler_tts.dac_wrapper.modeling_dac.DACModel'> is overwritten by shared audio_encoder config: DACConfig {
2025-04-13 13:22:25 pipeline-1 | "_name_or_path": "parler-tts/dac_44khZ_8kbps",
2025-04-13 13:22:25 pipeline-1 | "architectures": [
2025-04-13 13:22:25 pipeline-1 | "DACModel"
2025-04-13 13:22:25 pipeline-1 | ],
2025-04-13 13:22:25 pipeline-1 | "codebook_size": 1024,
2025-04-13 13:22:25 pipeline-1 | "frame_rate": 86,
2025-04-13 13:22:25 pipeline-1 | "latent_dim": 1024,
2025-04-13 13:22:25 pipeline-1 | "model_bitrate": 8,
2025-04-13 13:22:25 pipeline-1 | "model_type": "dac_on_the_hub",
2025-04-13 13:22:25 pipeline-1 | "num_codebooks": 9,
2025-04-13 13:22:25 pipeline-1 | "sampling_rate": 44100,
2025-04-13 13:22:25 pipeline-1 | "torch_dtype": "float32",
2025-04-13 13:22:25 pipeline-1 | "transformers_version": "4.46.1"
2025-04-13 13:22:25 pipeline-1 | }
2025-04-13 13:22:25 pipeline-1 |
2025-04-13 13:22:25 pipeline-1 | 2025-04-13 11:22:25,616 - parler_tts.modeling_parler_tts - WARNING - Config of the decoder: <class 'parler_tts.modeling_parler_tts.ParlerTTSForCausalLM'> is overwritten by shared decoder config: ParlerTTSDecoderConfig {
2025-04-13 13:22:25 pipeline-1 | "_name_or_path": "/fsx/yoach/tmp/artefacts/parler-tts-mini/decoder",
2025-04-13 13:22:25 pipeline-1 | "activation_dropout": 0.0,
2025-04-13 13:22:25 pipeline-1 | "activation_function": "gelu",
2025-04-13 13:22:25 pipeline-1 | "add_cross_attention": true,
2025-04-13 13:22:25 pipeline-1 | "architectures": [
2025-04-13 13:22:25 pipeline-1 | "ParlerTTSForCausalLM"
2025-04-13 13:22:25 pipeline-1 | ],
2025-04-13 13:22:25 pipeline-1 | "attention_dropout": 0.0,
2025-04-13 13:22:25 pipeline-1 | "bos_token_id": 1025,
2025-04-13 13:22:25 pipeline-1 | "codebook_weights": null,
2025-04-13 13:22:25 pipeline-1 | "cross_attention_implementation_strategy": null,
2025-04-13 13:22:25 pipeline-1 | "dropout": 0.1,
2025-04-13 13:22:25 pipeline-1 | "eos_token_id": 1024,
2025-04-13 13:22:25 pipeline-1 | "ffn_dim": 4096,
2025-04-13 13:22:25 pipeline-1 | "hidden_size": 1024,
2025-04-13 13:22:25 pipeline-1 | "initializer_factor": 0.02,
2025-04-13 13:22:25 pipeline-1 | "is_decoder": true,
2025-04-13 13:22:25 pipeline-1 | "layerdrop": 0.0,
2025-04-13 13:22:25 pipeline-1 | "max_position_embeddings": 4096,
2025-04-13 13:22:25 pipeline-1 | "model_type": "parler_tts_decoder",
2025-04-13 13:22:25 pipeline-1 | "num_attention_heads": 16,
2025-04-13 13:22:25 pipeline-1 | "num_codebooks": 9,
2025-04-13 13:22:25 pipeline-1 | "num_cross_attention_key_value_heads": 16,
2025-04-13 13:22:25 pipeline-1 | "num_hidden_layers": 24,
2025-04-13 13:22:25 pipeline-1 | "num_key_value_heads": 16,
2025-04-13 13:22:25 pipeline-1 | "pad_token_id": 1024,
2025-04-13 13:22:25 pipeline-1 | "rope_embeddings": false,
2025-04-13 13:22:25 pipeline-1 | "rope_theta": 10000.0,
2025-04-13 13:22:25 pipeline-1 | "scale_embedding": false,
2025-04-13 13:22:25 pipeline-1 | "tie_word_embeddings": false,
2025-04-13 13:22:25 pipeline-1 | "torch_dtype": "float32",
2025-04-13 13:22:25 pipeline-1 | "transformers_version": "4.46.1",
2025-04-13 13:22:25 pipeline-1 | "use_cache": true,
2025-04-13 13:22:25 pipeline-1 | "use_fused_lm_heads": false,
2025-04-13 13:22:25 pipeline-1 | "vocab_size": 1088
2025-04-13 13:22:25 pipeline-1 | }
2025-04-13 13:22:25 pipeline-1 |
2025-04-13 13:24:13 pipeline-1 | 2025-04-13 11:24:13,263 - TTS.parler_handler - INFO - Warming up ParlerTTSHandler
2025-04-13 13:24:13 pipeline-1 | The 'max_batch_size' argument of StaticCache is deprecated and will be removed in v4.46. Use the more precisely named 'batch_size' argument instead.
2025-04-13 13:24:13 pipeline-1 | 2025-04-13 11:24:13,434 - parler_tts.modeling_parler_tts - WARNING - `prompt_attention_mask` is specified but `attention_mask` is not. A full `attention_mask` will be created. Make sure this is the intended behaviour.
2025-04-13 13:24:13 pipeline-1 | Traceback (most recent call last):
2025-04-13 13:24:13 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 483, in <module>
2025-04-13 13:24:13 pipeline-1 | main()
2025-04-13 13:24:13 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 458, in main
2025-04-13 13:24:13 pipeline-1 | pipeline_manager = build_pipeline(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 282, in build_pipeline
2025-04-13 13:24:13 pipeline-1 | tts = get_tts_handler(module_kwargs, stop_event, lm_response_queue, send_audio_chunks_queue, should_listen, parler_tts_handler_kwargs, melo_tts_handler_kwargs, chat_tts_handler_kwargs, facebook_mms_tts_handler_kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 374, in get_tts_handler
2025-04-13 13:24:13 pipeline-1 | return ParlerTTSHandler(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/usr/src/app/baseHandler.py", line 20, in __init__
2025-04-13 13:24:13 pipeline-1 | self.setup(*setup_args, **setup_kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/usr/src/app/TTS/parler_handler.py", line 100, in setup
2025-04-13 13:24:13 pipeline-1 | self.warmup()
2025-04-13 13:24:13 pipeline-1 | File "/usr/src/app/TTS/parler_handler.py", line 158, in warmup
2025-04-13 13:24:13 pipeline-1 | _ = self.model.generate(**model_kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-04-13 13:24:13 pipeline-1 | return func(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/parler_tts/modeling_parler_tts.py", line 3564, in generate
2025-04-13 13:24:13 pipeline-1 | outputs = self._sample(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/transformers/generation/utils.py", line 3206, in _sample
2025-04-13 13:24:13 pipeline-1 | outputs = self(**model_inputs, return_dict=True)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
2025-04-13 13:24:13 pipeline-1 | return self._call_impl(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
2025-04-13 13:24:13 pipeline-1 | return forward_call(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 433, in _fn
2025-04-13 13:24:13 pipeline-1 | return fn(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 1116, in __call__
2025-04-13 13:24:13 pipeline-1 | return self._torchdynamo_orig_callable(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 472, in __call__
2025-04-13 13:24:13 pipeline-1 | return _compile(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_utils_internal.py", line 84, in wrapper_function
2025-04-13 13:24:13 pipeline-1 | return StrobelightCompileTimeProfiler.profile_compile_time(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time
2025-04-13 13:24:13 pipeline-1 | return func(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/contextlib.py", line 81, in inner
2025-04-13 13:24:13 pipeline-1 | return func(*args, **kwds)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 817, in _compile
2025-04-13 13:24:13 pipeline-1 | guarded_code = compile_inner(code, one_graph, hooks, transform)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 231, in time_wrapper
2025-04-13 13:24:13 pipeline-1 | r = func(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 636, in compile_inner
2025-04-13 13:24:13 pipeline-1 | out_code = transform_code_object(code, transform)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/bytecode_transformation.py", line 1185, in transform_code_object
2025-04-13 13:24:13 pipeline-1 | transformations(instructions, code_options)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 178, in _fn
2025-04-13 13:24:13 pipeline-1 | return fn(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 582, in transform
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2451, in run
2025-04-13 13:24:13 pipeline-1 | super().run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in CALL_FUNCTION_EX
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, argsvars.items, kwargsvars)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/nn_module.py", line 437, in call_function
2025-04-13 13:24:13 pipeline-1 | return tx.inline_user_function_return(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 749, in inline_user_function_return
2025-04-13 13:24:13 pipeline-1 | return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2666, in inline_call
2025-04-13 13:24:13 pipeline-1 | return cls.inline_call_(parent, func, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2782, in inline_call_
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in CALL_FUNCTION_EX
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, argsvars.items, kwargsvars)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 344, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2025-04-13 13:24:13 pipeline-1 | return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 749, in inline_user_function_return
2025-04-13 13:24:13 pipeline-1 | return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2666, in inline_call
2025-04-13 13:24:13 pipeline-1 | return cls.inline_call_(parent, func, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2782, in inline_call_
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2059, in CALL
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/nn_module.py", line 437, in call_function
2025-04-13 13:24:13 pipeline-1 | return tx.inline_user_function_return(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 749, in inline_user_function_return
2025-04-13 13:24:13 pipeline-1 | return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2666, in inline_call
2025-04-13 13:24:13 pipeline-1 | return cls.inline_call_(parent, func, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2782, in inline_call_
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in CALL_FUNCTION_EX
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, argsvars.items, kwargsvars)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 344, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2025-04-13 13:24:13 pipeline-1 | return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 749, in inline_user_function_return
2025-04-13 13:24:13 pipeline-1 | return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2666, in inline_call
2025-04-13 13:24:13 pipeline-1 | return cls.inline_call_(parent, func, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2782, in inline_call_
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2059, in CALL
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/nn_module.py", line 437, in call_function
2025-04-13 13:24:13 pipeline-1 | return tx.inline_user_function_return(
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 749, in inline_user_function_return
2025-04-13 13:24:13 pipeline-1 | return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2666, in inline_call
2025-04-13 13:24:13 pipeline-1 | return cls.inline_call_(parent, func, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2782, in inline_call_
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in CALL_FUNCTION_EX
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, argsvars.items, kwargsvars)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 344, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2025-04-13 13:24:13 pipeline-1 | return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 749, in inline_user_function_return
2025-04-13 13:24:13 pipeline-1 | return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2666, in inline_call
2025-04-13 13:24:13 pipeline-1 | return cls.inline_call_(parent, func, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2782, in inline_call_
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2059, in CALL
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 344, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2025-04-13 13:24:13 pipeline-1 | return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 749, in inline_user_function_return
2025-04-13 13:24:13 pipeline-1 | return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2666, in inline_call
2025-04-13 13:24:13 pipeline-1 | return cls.inline_call_(parent, func, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2782, in inline_call_
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2059, in CALL
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 344, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function
2025-04-13 13:24:13 pipeline-1 | return super().call_function(tx, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2025-04-13 13:24:13 pipeline-1 | return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 749, in inline_user_function_return
2025-04-13 13:24:13 pipeline-1 | return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2666, in inline_call
2025-04-13 13:24:13 pipeline-1 | return cls.inline_call_(parent, func, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2782, in inline_call_
2025-04-13 13:24:13 pipeline-1 | tracer.run()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 893, in run
2025-04-13 13:24:13 pipeline-1 | while self.step():
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 805, in step
2025-04-13 13:24:13 pipeline-1 | self.dispatch_table[inst.opcode](self, inst)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 499, in wrapper
2025-04-13 13:24:13 pipeline-1 | return inner_fn(self, inst)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2059, in CALL
2025-04-13 13:24:13 pipeline-1 | self.call_function(fn, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in call_function
2025-04-13 13:24:13 pipeline-1 | self.push(fn.call_function(self, args, kwargs))
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/misc.py", line 680, in call_function
2025-04-13 13:24:13 pipeline-1 | return self.obj.call_method(tx, self.name, args, kwargs)
2025-04-13 13:24:13 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/variables/misc.py", line 1093, in call_method
2025-04-13 13:24:13 pipeline-1 | unimplemented("Logger not supported for non-export cases")
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/_dynamo/exc.py", line 221, in unimplemented
2025-04-13 13:24:13 pipeline-1 | raise Unsupported(msg)
2025-04-13 13:24:13 pipeline-1 | torch._dynamo.exc.Unsupported: Logger not supported for non-export cases
2025-04-13 13:24:13 pipeline-1 |
2025-04-13 13:24:13 pipeline-1 | from user code:
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/parler_tts/modeling_parler_tts.py", line 2846, in forward
2025-04-13 13:24:13 pipeline-1 | decoder_outputs = self.decoder(
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
2025-04-13 13:24:13 pipeline-1 | return forward_call(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/parler_tts/modeling_parler_tts.py", line 1896, in forward
2025-04-13 13:24:13 pipeline-1 | outputs = self.model(
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
2025-04-13 13:24:13 pipeline-1 | return forward_call(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/parler_tts/modeling_parler_tts.py", line 1789, in forward
2025-04-13 13:24:13 pipeline-1 | decoder_outputs = self.decoder(
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
2025-04-13 13:24:13 pipeline-1 | return forward_call(*args, **kwargs)
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/parler_tts/modeling_parler_tts.py", line 1538, in forward
2025-04-13 13:24:13 pipeline-1 | causal_mask = self._update_causal_mask(
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/parler_tts/modeling_parler_tts.py", line 1696, in _update_causal_mask
2025-04-13 13:24:13 pipeline-1 | target_length = past_key_values.get_max_length()
2025-04-13 13:24:13 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/transformers/cache_utils.py", line 72, in get_max_length
2025-04-13 13:24:13 pipeline-1 | logger.warning_once(
2025-04-13 13:24:13 pipeline-1 |
2025-04-13 13:24:13 pipeline-1 | Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2025-04-13 13:24:13 pipeline-1 |
2025-04-13 13:24:13 pipeline-1 |
2025-04-13 13:24:13 pipeline-1 | You can suppress this exception and fall back to eager by setting:
2025-04-13 13:24:13 pipeline-1 | import torch._dynamo
2025-04-13 13:24:13 pipeline-1 | torch._dynamo.config.suppress_errors = True
2025-04-13 13:24:13 pipeline-1 |
2036-01-01 00:00:00
2001-01-01 00:00:00 xited with code 1
and when using melo, it crashes with this exception:
Click to expand
2025-04-13 13:09:01 pipeline-1 | 2025-04-13 11:09:01,469 - __main__ - ERROR - Error importing MeloTTSHandler. You might need to run: python -m unidic download
2025-04-13 13:09:01 pipeline-1 | Traceback (most recent call last):
2025-04-13 13:09:01 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/MeCab/__init__.py", line 137, in __init__
2025-04-13 13:09:01 pipeline-1 | super(Tagger, self).__init__(args)
2025-04-13 13:09:01 pipeline-1 | RuntimeError
2025-04-13 13:09:01 pipeline-1 |
2025-04-13 13:09:01 pipeline-1 | The above exception was the direct cause of the following exception:
2025-04-13 13:09:01 pipeline-1 |
2025-04-13 13:09:01 pipeline-1 | Traceback (most recent call last):
2025-04-13 13:09:01 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 483, in <module>
2025-04-13 13:09:01 pipeline-1 | main()
2025-04-13 13:09:01 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 458, in main
2025-04-13 13:09:01 pipeline-1 | pipeline_manager = build_pipeline(
2025-04-13 13:09:01 pipeline-1 | ^^^^^^^^^^^^^^^
2025-04-13 13:09:01 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 282, in build_pipeline
2025-04-13 13:09:01 pipeline-1 | tts = get_tts_handler(module_kwargs, stop_event, lm_response_queue, send_audio_chunks_queue, should_listen, parler_tts_handler_kwargs, melo_tts_handler_kwargs, chat_tts_handler_kwargs, facebook_mms_tts_handler_kwargs)
2025-04-13 13:09:01 pipeline-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-13 13:09:01 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 388, in get_tts_handler
2025-04-13 13:09:01 pipeline-1 | raise e
2025-04-13 13:09:01 pipeline-1 | File "/usr/src/app/s2s_pipeline.py", line 383, in get_tts_handler
2025-04-13 13:09:01 pipeline-1 | from TTS.melo_handler import MeloTTSHandler
2025-04-13 13:09:01 pipeline-1 | File "/usr/src/app/TTS/melo_handler.py", line 1, in <module>
2025-04-13 13:09:01 pipeline-1 | from melo.api import TTS
2025-04-13 13:09:01 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/melo/api.py", line 13, in <module>
2025-04-13 13:09:01 pipeline-1 | from . import utils
2025-04-13 13:09:01 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/melo/utils.py", line 13, in <module>
2025-04-13 13:09:01 pipeline-1 | from melo.text.cleaner import clean_text
2025-04-13 13:09:01 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/melo/text/cleaner.py", line 1, in <module>
2025-04-13 13:09:01 pipeline-1 | from . import chinese, japanese, english, chinese_mix, korean, french, spanish
2025-04-13 13:09:01 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/melo/text/japanese.py", line 367, in <module>
2025-04-13 13:09:01 pipeline-1 | _TAGGER = MeCab.Tagger()
2025-04-13 13:09:01 pipeline-1 | ^^^^^^^^^^^^^^
2025-04-13 13:09:01 pipeline-1 | File "/opt/conda/lib/python3.11/site-packages/MeCab/__init__.py", line 139, in __init__
2025-04-13 13:09:01 pipeline-1 | raise RuntimeError(error_info(rawargs)) from ee
2025-04-13 13:09:01 pipeline-1 | RuntimeError:
2025-04-13 13:09:01 pipeline-1 | ----------------------------------------------------------
2025-04-13 13:09:01 pipeline-1 |
2025-04-13 13:09:01 pipeline-1 | Failed initializing MeCab. Please see the README for possible solutions:
2025-04-13 13:09:01 pipeline-1 |
2025-04-13 13:09:01 pipeline-1 | https://github.com/SamuraiT/mecab-python3#common-issues
2025-04-13 13:09:01 pipeline-1 |
2025-04-13 13:09:01 pipeline-1 | If you are still having trouble, please file an issue here, and include the
2025-04-13 13:09:01 pipeline-1 | ERROR DETAILS below:
2025-04-13 13:09:01 pipeline-1 |
2025-04-13 13:09:01 pipeline-1 | https://github.com/SamuraiT/mecab-python3/issues
2025-04-13 13:09:01 pipeline-1 |
2025-04-13 13:09:01 pipeline-1 | issueを英語で書く必要はありません。
2025-04-13 13:09:01 pipeline-1 |
2025-04-13 13:09:01 pipeline-1 | ------------------- ERROR DETAILS ------------------------
2025-04-13 13:09:01 pipeline-1 | arguments:
2025-04-13 13:09:01 pipeline-1 | default dictionary path: /opt/conda/lib/python3.11/site-packages/unidic/dicdir
2025-04-13 13:09:01 pipeline-1 | [ifs] no such file or directory: /opt/conda/lib/python3.11/site-packages/unidic/dicdir/mecabrc
2025-04-13 13:09:01 pipeline-1 | ----------------------------------------------------------
2025-04-13 13:09:01 pipeline-1 |
2036-01-01 00:00:00
2001-01-01 00:00:00 xited with code 1