[Usage] ImportError: cannot import name 'LlavaLlamaForCausalLM' from 'llava.model'

[TobiasJu]

Describe the issue

Issue:
So i updated the repo and now i can not start the server anymore. I deleted the repo and cloned it again, but get the same error.

System:
Win10 WSL2 Ubuntu
pip 24.0 from /home/tobias/.local/lib/python3.10/site-packages/pip (python 3.10)

Command:

git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
conda create --name LLaVA python=3.10.12
conda activate LLaV
pip install -e .
pip install flash-attn --no-build-isolation
python3 -m llava.serve.controller --host 0.0.0.0 --port 10000

Log:

python3 -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-v1.5-13b --load-4bit
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/mnt/a/KI/LLaVA/llava/__init__.py", line 1, in <module>
    from .model import LlavaLlamaForCausalLM
ImportError: cannot import name 'LlavaLlamaForCausalLM' from 'llava.model' (/mnt/a/KI/LLaVA/llava/model/__init__.py)

Packages:

pip list
Package                   Version       Editable project location
------------------------- ------------- -------------------------
accelerate                0.21.0
aiofiles                  23.2.1
aiohttp                   3.9.3
aiosignal                 1.3.1
altair                    5.2.0
annotated-types           0.6.0
anyio                     4.2.0
async-timeout             4.0.3
attrs                     23.2.0
bitsandbytes              0.41.0
blinker                   1.4
CacheControl              0.12.10
cachy                     0.3.0
certifi                   2023.11.17
chardet                   4.0.0
charset-normalizer        3.3.2
cleo                      0.8.1
click                     8.1.7
clikit                    0.6.2
cmake                     3.28.1
colorama                  0.4.6
command-not-found         0.3
contourpy                 1.2.0
crashtest                 0.3.1
cryptography              3.4.8
cycler                    0.12.1
dbus-python               1.2.18
distlib                   0.3.4
distro                    1.7.0
distro-info               1.1+ubuntu0.2
einops                    0.6.1
einops-exts               0.0.4
exceptiongroup            1.2.0
fastapi                   0.109.0
ffmpy                     0.3.1
filelock                  3.13.1
flash-attn                2.5.2
fonttools                 4.47.2
frozenlist                1.4.1
fsspec                    2023.12.2
gradio                    4.16.0
gradio_client             0.8.1
h11                       0.14.0
html5lib                  1.1
httpcore                  0.17.3
httplib2                  0.20.2
httpx                     0.24.0
huggingface-hub           0.20.3
idna                      3.6
importlib-metadata        4.6.4
importlib-resources       6.1.1
jeepney                   0.7.1
Jinja2                    3.1.3
joblib                    1.3.2
jsonschema                4.21.1
jsonschema-specifications 2023.12.1
keyring                   23.5.0
kiwisolver                1.4.5
launchpadlib              1.10.16
lazr.restfulclient        0.14.4
lazr.uri                  1.0.6
linkify-it-py             2.0.2
lit                       17.0.6
llava                     1.2.2.post1   /mnt/a/KI/LLaVA
lockfile                  0.12.2
markdown-it-py            2.2.0
markdown2                 2.4.12
MarkupSafe                2.1.4
matplotlib                3.8.2
mdit-py-plugins           0.3.3
mdurl                     0.1.2
more-itertools            8.10.0
mpmath                    1.3.0
msgpack                   1.0.3
multidict                 6.0.4
netifaces                 0.11.0
networkx                  3.2.1
ninja                     1.11.1.1
numpy                     1.26.3
nvidia-cublas-cu11        11.10.3.66
nvidia-cublas-cu12        12.1.3.1
nvidia-cuda-cupti-cu11    11.7.101
nvidia-cuda-cupti-cu12    12.1.105
nvidia-cuda-nvrtc-cu11    11.7.99
nvidia-cuda-nvrtc-cu12    12.1.105
nvidia-cuda-runtime-cu11  11.7.99
nvidia-cuda-runtime-cu12  12.1.105
nvidia-cudnn-cu11         8.5.0.96
nvidia-cudnn-cu12         8.9.2.26
nvidia-cufft-cu11         10.9.0.58
nvidia-cufft-cu12         11.0.2.54
nvidia-curand-cu11        10.2.10.91
nvidia-curand-cu12        10.3.2.106
nvidia-cusolver-cu11      11.4.0.1
nvidia-cusolver-cu12      11.4.5.107
nvidia-cusparse-cu11      11.7.4.91
nvidia-cusparse-cu12      12.1.0.106
nvidia-nccl-cu11          2.14.3
nvidia-nccl-cu12          2.18.1
nvidia-nvjitlink-cu12     12.3.101
nvidia-nvtx-cu11          11.7.91
nvidia-nvtx-cu12          12.1.105
oauthlib                  3.2.0
orjson                    3.9.12
packaging                 23.2
pandas                    2.2.0
pastel                    0.2.1
peft                      0.4.0
pexpect                   4.8.0
pillow                    10.2.0
pip                       24.0
pkginfo                   1.8.2
platformdirs              2.5.1
poetry-core               1.0.7
psutil                    5.9.8
ptyprocess                0.7.0
pydantic                  2.6.1
pydantic_core             2.16.2
pydub                     0.25.1
Pygments                  2.17.2
PyGObject                 3.42.1
PyJWT                     2.3.0
pylev                     1.2.0
pyparsing                 2.4.7
python-apt                2.4.0+ubuntu2
python-dateutil           2.8.2
python-multipart          0.0.6
pytz                      2023.4
PyYAML                    5.4.1
referencing               0.33.0
regex                     2023.12.25
requests                  2.31.0
requests-toolbelt         0.9.1
rich                      13.7.0
rpds-py                   0.17.1
ruff                      0.2.1
safetensors               0.4.2
scikit-learn              1.2.2
scipy                     1.12.0
SecretStorage             3.3.1
semantic-version          2.10.0
sentencepiece             0.1.99
setuptools                59.6.0
shellingham               1.4.0
shortuuid                 1.0.11
six                       1.16.0
sniffio                   1.3.0
starlette                 0.35.1
svgwrite                  1.4.3
sympy                     1.12
systemd-python            234
threadpoolctl             3.2.0
timm                      0.6.13
tokenizers                0.15.1
tomlkit                   0.12.0
toolz                     0.12.1
torch                     2.1.2
torchvision               0.16.2
tqdm                      4.66.1
transformers              4.37.2
triton                    2.1.0
typer                     0.9.0
typing_extensions         4.9.0
tzdata                    2023.4
ubuntu-advantage-tools    8001
uc-micro-py               1.0.2
ufw                       0.36.1
unattended-upgrades       0.1
urllib3                   2.2.0
uvicorn                   0.27.0.post1
virtualenv                20.13.0+ds
wadllib                   1.3.6
wavedrom                  2.0.3.post3
webencodings              0.5.1
websockets                11.0.3
wheel                     0.37.1
yarl                      1.9.4
zipp                      1.0.0

Screenshots:

[anubhavashok]

Try importing the packages without the "try, except" block for a more informative error.

Probably related to flash attn installation.

In my case, the following worked:

pip uninstall flash-attn

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

pip install flash-attn --no-build-isolation --no-cache-dir

[pseudotensor]

For me it was problem with packages like deepspeed etc. that led to above. But any issue will lead to it. This is the combo that worked for me:

# CUDA_HOME may not be enough:
cd /usr/local/
sudo rm -rf /usr/local/cuda # if link only
sudo ln -s /usr/local/cuda-12.1 cuda

export CUDA_HOME=/usr/local/cuda-12.1
export PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cu121"

conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip  # enable PEP 660 support

git clone https://github.com/haotian-liu/LLaVA.git llava
cd llava

pip install -e .
pip install -e ".[train]"
pip install torch==2.1.2 torchvision==0.16.2 triton==2.1.0 accelerate==0.26.1 deepspeed==0.13.1 pynvml==11.5.0 --upgrade
pip install "sglang[all]"
pip install flash-attn==2.5.2 --no-build-isolation

[celll1]

I am also having the same error.
I have found that this error occurs depending on the order in which the packages are installed, but have not been able to determine the cause.
If I create a new environment and then install only LLaVA, it works correctly, but I would like to use it in combination with other modules, so I would like to see this bug fixed.

[TobiasJu]

I got rid of the try-except block in llava/model/__init__.py and then reinstalled from root with pip install -e . and the error went away for me

before:

try:
    from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig
    from .language_model.llava_mpt import LlavaMptForCausalLM, LlavaMptConfig
    from .language_model.llava_mistral import LlavaMistralForCausalLM, LlavaMistralConfig
except:
    pass

after:

from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig
from .language_model.llava_mpt import LlavaMptForCausalLM, LlavaMptConfig
from .language_model.llava_mistral import LlavaMistralForCausalLM, LlavaMistralConfig

Tried and now i get this error:

python3 -m llava.serve.controller --host 0.0.0.0 --port 10000
[2024-02-08 14:52:59,501] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/tobias/.local/lib/python3.10/site-packages/pydantic/_internal/_config.py:322: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
* 'validate_all' has been renamed to 'validate_default'
  warnings.warn(message, UserWarning)
/home/tobias/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_persistence_threshold" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
Traceback (most recent call last):
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1364, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/generation/utils.py", line 28, in <module>
    from ..integrations.deepspeed import is_deepspeed_zero3_enabled
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/integrations/deepspeed.py", line 49, in <module>
    from accelerate.utils.deepspeed import HfDeepSpeedConfig as DeepSpeedConfig
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 133, in <module>
    from .launch import (
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/utils/launch.py", line 33, in <module>
    from ..utils.other import is_port_in_use, merge_dicts
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/utils/other.py", line 30, in <module>
    from deepspeed import DeepSpeedEngine
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/__init__.py", line 16, in <module>
    from . import module_inject
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/module_inject/__init__.py", line 6, in <module>
    from .replace_module import replace_transformer_layer, revert_transformer_layer, ReplaceWithTensorSlicing, GroupQuantizer, generic_injection
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 792, in <module>
    from ..pipe import PipelineModule
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/pipe/__init__.py", line 6, in <module>
    from ..runtime.pipe import PipelineModule, LayerSpec, TiedLayerSpec
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/pipe/__init__.py", line 6, in <module>
    from .module import PipelineModule, LayerSpec, TiedLayerSpec
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/pipe/module.py", line 19, in <module>
    from ..activation_checkpointing import checkpointing
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 25, in <module>
    from deepspeed.runtime.config import DeepSpeedConfig
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 29, in <module>
    from .zero.config import get_zero_config, ZeroStageEnum
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/__init__.py", line 6, in <module>
    from .partition_parameters import ZeroParamType
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 616, in <module>
    class Init(InsertPostInitMethodToModuleSubClasses):
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 618, in Init
    param_persistence_threshold = get_config_default(DeepSpeedZeroConfig, "param_persistence_threshold")
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/config_utils.py", line 116, in get_config_default
    field_name).required, f"'{field_name}' is a required field and does not have a default value"
AttributeError: 'FieldInfo' object has no attribute 'required'. Did you mean: 'is_required'?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1364, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 40, in <module>
    from ...modeling_utils import PreTrainedModel
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 44, in <module>
    from .generation import GenerationConfig, GenerationMixin
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1354, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1366, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
'FieldInfo' object has no attribute 'required'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/mnt/a/KI/LLaVA/llava/__init__.py", line 1, in <module>
    from .model import LlavaLlamaForCausalLM
  File "/mnt/a/KI/LLaVA/llava/model/__init__.py", line 2, in <module>
    from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig
  File "/mnt/a/KI/LLaVA/llava/model/language_model/llava_llama.py", line 21, in <module>
    from transformers import AutoConfig, AutoModelForCausalLM, \
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1355, in __getattr__
    value = getattr(module, name)
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1354, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1366, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
'FieldInfo' object has no attribute 'required'

[pseudotensor]

@TobiasJu I got same, but then did what I did:

#1101 (comment)

[haotian-liu]

We find that this is due to flash-attn compiled previously with a different version of pytorch. Please reinstall that with:

pip install flash-attn --no-build-isolation --no-cache-dir

[pseudotensor]

Correct, but even that isn't enough, deepspeed etc. have to be correct range of versions, else hit the other issues shown above.

[haotian-liu]

Thanks, deepspeed==0.12.6 works for me in my local setting, so basically I found that if you need to train the model, running the below command is sufficient?

pip install -e ".[train]"
pip install flash-attn --no-build-isolation --no-cache-dir

[pseudotensor]

In my and the above case the deepspeed error was hit with inference too.

I didn't check exactly which deepspeed version was required. I just did pip install accelerate deepspeed --upgrade and it started working, so I wanted to remember the versions that worked so wrote those down.

[haotian-liu]

Great, thank you for the information!

[nitzanguetta]

After following the instructions in the repo, I encountered the same error:

from .model import LlavaLlamaForCausalLM
ImportError: cannot import name 'LlavaLlamaForCausalLM' from 'llava.model' (/mnt/a/KI/LLaVA/llava/model/__init__.py)

Currently, after attempting #1101 (comment), I got this one:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yonatanbitton/experiments/llava/llava/model/__init__.py", line 8, in <module>
    from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig
  File "/home/yonatanbitton/experiments/llava/llava/model/language_model/llava_llama.py", line 21, in <module>
    from transformers import AutoConfig, AutoModelForCausalLM, \
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/opt/conda/envs/llava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1355, in __getattr__
    value = getattr(module, name)
  File "/opt/conda/envs/llava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1354, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/opt/conda/envs/llava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1366, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
libcudart.so.11.0: cannot open shared object file: No such file or directory

All attempts to resolve this issue have only resulted in more errors.

[ Debian GNU/Linux 11 (bullseye)
pip 24.0, python 3.10
(deepspeed==0.13.1/0.12.6, cuda==12.1) ]

[pseudotensor]

I got that libcudart.so.12.0: cannot open shared object file: No such file or directory type errors, but that's just because you must not have set CUDA_HOME corretly or do not have cuda toolkit installed for (there) cuda 12.1. It shouldn't be looking for cuda 11 if you really freshly installed with CUDA_HOME and that PIP stuff set.

A last thing I may have done is to literally relink the cuda directory:

cd /usr/local/
sudo rm -rf /usr/local/cuda # if link only
sudo ln -s /usr/local/cuda-12.1 cuda

But make sure all your cuda stuff is consistent.

[Guruprasad68]

#1101 (comment)
It will be great if the installation steps can be updated with this.

[dancasas]

Seems pretty random, but I got rid of this error by just commenting out the only line in LLaVA/llava/__init__.py

#from .model import LlavaLlamaForCausalLM

Now my installation is running correctly. Can anybody explain?

[lukaemon]

Same problem here.

Thanks, deepspeed==0.12.6 works for me in my local setting, so basically I found that if you need to train the model, running the below command is sufficient?

pip install -e ".[train]"
pip install flash-attn --no-build-isolation --no-cache-dir

[zengxingchen]

pip uninstall  flash-attn
pip install -e ".[train]"
pip install flash-attn --no-build-isolation --no-cache-dir

The above order worked for me. The order is important.

[sucongCJS]

Seems pretty random, but I got rid of this error by just commenting out the only line in LLaVA/llava/__init__.py

#from .model import LlavaLlamaForCausalLM

Now my installation is running correctly. Can anybody explain?

I tried but,
NameError: name 'LlavaLlamaForCausalLM' is not defined

[nrikoh]

In my and the above case the deepspeed error was hit with inference too.

I didn't check exactly which deepspeed version was required. I just did pip install accelerate deepspeed --upgrade and it started working, so I wanted to remember the versions that worked so wrote those down.

this works for me ,thanks!

[20191864218]

I am also having the same error. I have found that this error occurs depending on the order in which the packages are installed, but have not been able to determine the cause. If I create a new environment and then install only LLaVA, it works correctly, but I would like to use it in combination with other modules, so I would like to see this bug fixed.

I'm also encountering this issue. Have you resolved it?

[frankRenlf]

'''
pip uninstall flash-attn
pip install -e ".[train]"
pip install flash-attn --no-build-isolation --no-cache-dir
'''
this works, many thanks

[aymenabid-lab]

fro me it don't work!
(llava) C:\Users\aymen\LLaVA>pip uninstall flash-attn
WARNING: Skipping flash-attn as it is not installed.

(llava) C:\Users\aymen\LLaVA>pip install -e ".[train]"
gives:
Collecting deepspeed==0.12.6 (from llava==1.2.2.post1)
Downloading deepspeed-0.12.6.tar.gz (1.2 MB)
---------------------------------------- 1.2/1.2 MB 1.4 MB/s eta 0:00:00
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [15 lines of output]
test.c
LINK : fatal error LNK1181: cannot open input file 'aio.lib'
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "C:\Users\aymen\AppData\Local\Temp\pip-install-554dhcbc\deepspeed_766af2b6636b428d9bb97f1b3acb1da1\setup.py", line 182, in
abort(f"Unable to pre-compile {op_name}")
File "C:\Users\aymen\AppData\Local\Temp\pip-install-554dhcbc\deepspeed_766af2b6636b428d9bb97f1b3acb1da1\setup.py", line 52, in abort
assert False, msg
AssertionError: Unable to pre-compile async_io
DS_BUILD_OPS=1
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[WARNING] One can disable async_io with DS_BUILD_AIO=0
[ERROR] Unable to pre-compile async_io
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

And
pip install flash-attn --no-build-isolation --no-cache-dir
gives
Building wheels for collected packages: flash-attn
Building wheel for flash-attn (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [215 lines of output]
fatal: not a git repository (or any of the parent directories): .git

  torch.__version__  = 2.1.2+cu121


  C:\ProgramData\anaconda3\envs\llava\lib\site-packages\setuptools\__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************

  !!
    dist.fetch_build_eggs(dist.setup_requires)
  running bdist_wheel
  Guessing wheel URL:  https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.6/flash_attn-2.5.6+cu122torch2.1cxx11abiFALSE-cp310-cp310-win_amd64.whl
  Precompiled wheel not found. Building from source...
  C:\ProgramData\anaconda3\envs\llava\lib\site-packages\torch\utils\cpp_extension.py:502: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
    warnings.warn(msg.format('we could not find ninja.'))

[patrick-tssn]

I got rid of the try-except block in llava/model/__init__.py and then reinstalled from root with pip install -e . and the error went away for me
before:

try:
    from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig
    from .language_model.llava_mpt import LlavaMptForCausalLM, LlavaMptConfig
    from .language_model.llava_mistral import LlavaMistralForCausalLM, LlavaMistralConfig
except:
    pass

after:

from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig
from .language_model.llava_mpt import LlavaMptForCausalLM, LlavaMptConfig
from .language_model.llava_mistral import LlavaMistralForCausalLM, LlavaMistralConfig

Tried and now i get this error:

python3 -m llava.serve.controller --host 0.0.0.0 --port 10000
[2024-02-08 14:52:59,501] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/tobias/.local/lib/python3.10/site-packages/pydantic/_internal/_config.py:322: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
* 'validate_all' has been renamed to 'validate_default'
  warnings.warn(message, UserWarning)
/home/tobias/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_persistence_threshold" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
Traceback (most recent call last):
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1364, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/generation/utils.py", line 28, in <module>
    from ..integrations.deepspeed import is_deepspeed_zero3_enabled
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/integrations/deepspeed.py", line 49, in <module>
    from accelerate.utils.deepspeed import HfDeepSpeedConfig as DeepSpeedConfig
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 133, in <module>
    from .launch import (
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/utils/launch.py", line 33, in <module>
    from ..utils.other import is_port_in_use, merge_dicts
  File "/home/tobias/.local/lib/python3.10/site-packages/accelerate/utils/other.py", line 30, in <module>
    from deepspeed import DeepSpeedEngine
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/__init__.py", line 16, in <module>
    from . import module_inject
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/module_inject/__init__.py", line 6, in <module>
    from .replace_module import replace_transformer_layer, revert_transformer_layer, ReplaceWithTensorSlicing, GroupQuantizer, generic_injection
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 792, in <module>
    from ..pipe import PipelineModule
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/pipe/__init__.py", line 6, in <module>
    from ..runtime.pipe import PipelineModule, LayerSpec, TiedLayerSpec
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/pipe/__init__.py", line 6, in <module>
    from .module import PipelineModule, LayerSpec, TiedLayerSpec
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/pipe/module.py", line 19, in <module>
    from ..activation_checkpointing import checkpointing
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 25, in <module>
    from deepspeed.runtime.config import DeepSpeedConfig
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 29, in <module>
    from .zero.config import get_zero_config, ZeroStageEnum
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/__init__.py", line 6, in <module>
    from .partition_parameters import ZeroParamType
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 616, in <module>
    class Init(InsertPostInitMethodToModuleSubClasses):
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 618, in Init
    param_persistence_threshold = get_config_default(DeepSpeedZeroConfig, "param_persistence_threshold")
  File "/home/tobias/.local/lib/python3.10/site-packages/deepspeed/runtime/config_utils.py", line 116, in get_config_default
    field_name).required, f"'{field_name}' is a required field and does not have a default value"
AttributeError: 'FieldInfo' object has no attribute 'required'. Did you mean: 'is_required'?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1364, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 40, in <module>
    from ...modeling_utils import PreTrainedModel
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 44, in <module>
    from .generation import GenerationConfig, GenerationMixin
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1354, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1366, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
'FieldInfo' object has no attribute 'required'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/mnt/a/KI/LLaVA/llava/__init__.py", line 1, in <module>
    from .model import LlavaLlamaForCausalLM
  File "/mnt/a/KI/LLaVA/llava/model/__init__.py", line 2, in <module>
    from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig
  File "/mnt/a/KI/LLaVA/llava/model/language_model/llava_llama.py", line 21, in <module>
    from transformers import AutoConfig, AutoModelForCausalLM, \
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1355, in __getattr__
    value = getattr(module, name)
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1354, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/tobias/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1366, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
'FieldInfo' object has no attribute 'required'

I believe that in certain situations, it would be beneficial to disable the try-except mechanism during the debugging process, particularly after making modifications to the original codebase. For instance, in this scenario, the error arises due to the binding of the local package within the Conda environment, so you can see this for help.

[jam-cc]

In my and the above case the deepspeed error was hit with inference too.

I didn't check exactly which deepspeed version was required. I just did pip install accelerate deepspeed --upgrade and it started working, so I wanted to remember the versions that worked so wrote those down.

It works for me, thanks!

[CHELSEA234]

A quick update. I followed the installment routine and attended the discussion on this page, but I still ran into the problem.
I noticed one potential reason is the torch version, and then I followed this link to get rid of the problem:
pytorch/pytorch#111469 (comment)

[ZhengShuozai]

During the training process, I always encountered this problem. Later, based on my CUDA=11.7, I downgraded the torch version and also downgraded the flash attn version, and the problem was solved.
Currently torch=1.13.1 flash-attn==2.3 tokenizers == 0.11.4

[pervaizniazi]

I solved the issue by changing Python version:

replacing
conda create -n llava python=3.10 -y
with
conda create -n llava python=3.9 -y

[liucheny]

@pseudotensor Does CUDA have to be 12.1？

[boyugou]

@pseudotensor Does CUDA have to be 12.1？

same question

[boyugou]

OK. I found a more general solution to this problem:

Updating the package requirement to:

"torch", "torchvision","transformers==4.37.2","accelerate==0.28.0","deepspeed==0.14.4"

So basically, the error is caused by a lot of package incompatibility. The try-except in the init has hidden the real issues. But the issues have been actually solved by updated deepspeed and accelerate.

Hope this will be useful for people like me who have to switch to newer torch/CUDA/NCCL due to the constraints of hardware.

BTW, probably this codebase also requires flash-attn<=2.6.3

[liuxiang09]

OK. I found a more general solution to this problem:

Updating the package requirement to:

"torch", "torchvision","transformers==4.37.2","accelerate==0.28.0","deepspeed==0.14.4"

So basically, the error is caused by a lot of package incompatibility. The try-except in the init has hidden the real issues. But the issues has been actually solved by updated deepspeed and accelerate.

Hope this will be useful for people like me who have to switch to newer torch/CUDA/NCCL due to the constraints of hardware.

BTW, probably this codebase also requires flash-attn<=2.6.3

thank you ! you actually solved this porblem and it's very useful.

[boyugou]

OK. I found a more general solution to this problem:
Updating the package requirement to:
"torch", "torchvision","transformers==4.37.2","accelerate==0.28.0","deepspeed==0.14.4"
So basically, the error is caused by a lot of package incompatibility. The try-except in the init has hidden the real issues. But the issues has been actually solved by updated deepspeed and accelerate.
Hope this will be useful for people like me who have to switch to newer torch/CUDA/NCCL due to the constraints of hardware.
BTW, probably this codebase also requires flash-attn<=2.6.3

thank you ! you actually solved this porblem and it's very useful.

hh. You’re welcome. I had to solve the problem due to some constraints of our infra at the time.