ImportError: /ramyapra/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol:

[ramyaprabhu-alt]

I'm trying to run vllm and lm-eval-harness. I'm using vllm 0.2.5. After I'm done installing both, if I try importing vllm I get the following error:
File "/ramyapra/lm-evaluation-harness/lm_eval/models/__init__.py", line 7, in <module> from . import vllm_causallms File "/ramyapra/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 16, in <module> from vllm import LLM, SamplingParams File "/ramyapra/vllm/vllm/__init__.py", line 3, in <module> from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs File "/ramyapra/vllm/vllm/engine/arg_utils.py", line 6, in <module> from vllm.config import (CacheConfig, ModelConfig, ParallelConfig, File "/ramyapra/vllm/vllm/config.py", line 9, in <module> from vllm.utils import get_cpu_memory, is_hip File "/ramyapra/vllm/vllm/utils.py", line 8, in <module> from vllm._C import cuda_utils ImportError: /ramyapra/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops19empty_memory_format4callEN3c108ArrayRefINS2_6SymIntEEESt8optionalINS2_10ScalarTypeEES6_INS2_6LayoutEES6_INS2_6DeviceEES6_IbES6_INS2_12MemoryFormatEE

I'm using the NGC docker container 23:10-py3.

[sudarshan-kamath]

Uninstall the package called transformer-engine by using the command pip uninstall transformer-engine. Install transformers and accelerate if not already installed. Then it should work.

[bfotzo]

Uninstall the package called transformer-engine by using the command pip uninstall transformer-engine. Install transformers and accelerate if not already installed. Then it should work.

I tried this but didn't work

[sudarshan-kamath]

Please post the vllm version and then the steps to reproduce this. Which model are you using? Cuda version on the system and the docker (if you use it).

[hassan-twelvelabs]

I am also experiencing this issue.
env: vllm-0.2.2 , cuda 12.2 , no docker. Local build of vllm from source, then running examples/offline_inference.py.

[choco9966]

i also experience
vllm==0.2.5 / accelerate==0.27.2 / transformers==4.37.2

[sylcjl]

Any idea? I am also facing this issue
Docker image: NGC docker container 23.10-py3 too.

• vllm==0.3.1+cu122 (Build from source)
• transformers==4.35.2
• transformer-engine==1.4.0.dev0
• accelerate==0.27.0

[rkuo2000]

same problem, I am running on Kaggle.com

Successfully installed aioprometheus-23.12.0 cupy-cuda12x-12.1.0 pynvml-11.5.0 quantile-python-1.1 transformers-4.38.1 triton-2.2.0 vllm-0.3.2 xformers-0.0.23.post1

[sylcjl]

It seems to be PyTorch isn't working with Cuda-12.2 on NGC-PyTorch 23.10-py3.
I fix this issue on NGC-PyTorch 23.07-py3 and reinstall pytorch with v2.1.2

[mohammad-yousuf]

same issue. anyone able to fix?

cuda: 12.0.1
torch : 2.2.1
transformers: 4.38.2
vllm: 0.3.2
accelerate: 0.22.0

UPDATE: solved by downgrading torch to 2.1.2

[surak]

#2797 is the same

[RylanSchaeffer]

I'm hitting this same problem :(

[RylanSchaeffer]

@sudarshan-kamath , pip uninstall transformer-engine also did not work for me

[youkaichao]

@RylanSchaeffer can you try to installing a matching version of pytorch with vllm?

e.g. vllm 0.3.3 with pytorch 2.1 .

[RylanSchaeffer]

Versions:

>>> import torch
>>> torch.__version__
'2.1.2.post301'
>>> import vllm
>>> vllm.__version__
'0.3.3'

Error:

ImportError: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/site-packages/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKSs

[RylanSchaeffer]

@youkaichao , here's a script I'm using to debug:

from vllm import LLM, SamplingParams

sampling_params = SamplingParams(max_tokens=50, temperature=1.0)

llm = LLM(
    model=f"gpt2",
    tensor_parallel_size=4,
    # load_format="pt",
)

outputs = llm.generate(
    ["My name is Lewis and I like to", "Today is a good day to", "John has"],
    sampling_params,
)


for output in outputs:
    print(output.outputs[0].text)
    print("=========")