[bitsandbytes]: support read bnb pre-quantized model

[thesues]

huggingface is bitsandbytes pre quantized model such as

• lllyasviel/omost-llama-3-8b-4bit
• unsloth/tinyllama-bnb-4bit
  ...

this will support these pre quantized for vllm

@chenqianfzh @Yard1 @jeejeelee

[chenqianfzh]

@Yard1 @jeejeelee @mgoin

The changes from @thesues improves my previous work on QLoRA & Bnb (#4776). It now supports models whose weights are published as bnb-quantized.

Other than that, it also cleaned my previous code and fixed a bug ( the previous version will run into error in scenarios such as GQA).

The changes looks good to me. Could you also take a look?

[mgoin]

I'm not fond of the term "prequant" here, could it be something along the lines of "quantized_checkpoint"?

[thesues]

sure

[mgoin]

What isn't supported about it? It seems like no exception is being thrown

[thesues]

a typo here, it should be "only quant_state.bitsandbytes__nf4 is supported". other libraries such as hf transformer supports quant_state.bitsandbytes__fp4.

[mgoin]

Ditto on pre_quant, it should be talking about the checkpoint being quantized

[mgoin]

Appreciate the improvements and ability to natively load! I think it would be great to followup with a documentation page in the quantization section to show how to deploy bnb models directly in vLLM, perhaps straight from a quick finetune in unsloth.

[thesues]

Appreciate the improvements and ability to natively load! I think it would be great to followup with a documentation page in the quantization section to show how to deploy bnb models directly in vLLM, perhaps straight from a quick finetune in unsloth.

sure, I added a very simple bnb.rst in docs

[thesues]

@mgoin can you review this version?

[QwertyJack]

Redundant commma.

[vrdn-23]

Is this PR good to merge now? It would be really great if we could get it in before the next scheduled release (#6433)!
cc @mgoin @QwertyJack

[QwertyJack]

Is this PR good to merge now? It would be really great if we could get it in before the next scheduled release (#6433)! cc @mgoin @QwertyJack

SGTM

[thesues]

Is this PR good to merge now? It would be really great if we could get it in before the next scheduled release (#6433)! cc @mgoin @QwertyJack

SGTM

is there anything I could do to improve this patch?

[eliasecchig]

Any ETA for this feature to be merged? Really keen to use it

[mgoin]

Thanks for pinging, LGTM with a small docs fix

[QwertyJack]

Is this PR good to merge now? It would be really great if we could get it in before the next scheduled release (#6433)! cc @mgoin @QwertyJack

SGTM

As I mentioned earlier, Look Good To Me!

[njhill]

@thesues do you think you could resolve the new conflicts?

[Maxusmusti]

I've just tried installing from this PR's branch and testing it out, this solution worked for me!

python3 -m vllm.entrypoints.openai.api_server --model <bnb_4bit_model> --quantization bitsandbytes --load-format bitsandbytes

used to produce issues like:
ValueError: Cannot find any of ['adapter_name_or_path'] in the model's quantization config.
and once args were added:
KeyError: 'model.layers.0.self_attn.qkv_proj.weight'

but now model loading seems to work great:

WARNING 07-23 16:35:07 config.py:218] bitsandbytes quantization is not fully optimized yet. The speed can be slower than non-quantized models.
...
INFO 07-23 16:35:08 loader.py:800] Loading weights with BitsAndBytes quantization.  May take a while ...
INFO 07-23 16:35:10 model_runner.py:164] Loading model weights took 3.6494 GB
INFO 07-23 16:35:11 gpu_executor.py:83] # GPU blocks: 1977, # CPU blocks: 512
...
INFO:     Started server process [15386]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000/ (Press CTRL+C to quit)

[mgoin]

It would be great if someone could validate that the new Llama 3.1 8B BNB checkpoint loads: https://huggingface.co/hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4

[thesues]

It would be great if someone could validate that the new Llama 3.1 8B BNB checkpoint loads: https://huggingface.co/hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4

Meta-Llama-3.1-8B-Instruct-BNB-NF4 works as expected.

from vllm import LLM, SamplingParams


model_id = "hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4"
llm = LLM(model=model_id, trust_remote_code=True, enforce_eager=True, quantization="bitsandbytes", load_format="bitsandbytes", m
ax_model_len=4096)
sampling_params = SamplingParams(max_tokens=20)

outputs = llm.generate("The capital of United States is", sampling_params)
for output in outputs:
    print(output.outputs[0].text)

logs:

WARNING 07-23 21:01:21 config.py:246] bitsandbytes quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 07-23 21:01:21 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4', speculative_config=None, tokenizer='hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.BITSANDBYTES, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=bitsandbytes, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-23 21:01:22 model_runner.py:680] Starting to load model hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4...
INFO 07-23 21:01:22 loader.py:821] Loading weights with BitsAndBytes quantization.  May take a while ...
INFO 07-23 21:01:22 weight_utils.py:224] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  5.58it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.49it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.68it/s]

Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  5.56it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.48it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.66it/s]

INFO 07-23 21:01:25 model_runner.py:692] Loading model weights took 5.3424 GB
INFO 07-23 21:01:27 gpu_executor.py:102] # GPU blocks: 6603, # CPU blocks: 2048
Processed prompts: 100%|█████████████████████| 1/1 [00:01<00:00,  1.57s/it, est. speed input: 3.83 toks/s, output: 25.52 toks/s]
 Washington, D.C.

[thesues]

@thesues do you think you could resolve the new conflicts?

done

[mgoin]

Thanks a lot for testing further and sticking with this, LGTM!