[Model] Add BNB quantization support for Idefics3

[B-201]

Add BnB quantization support for Idefics3.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @B-201.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jeejeelee]

Please check and add prefix for the remaining modules, thanks

[jeejeelee]

LGTM, @DarkLight1337 could you plz look at this PR?
BTW, @B-201 this model is not tested in the current CI, I suggest you provide your local test results

[B-201]

LGTM, @DarkLight1337 could you plz look at this PR? BTW, @B-201 this model is not tested in the current CI, I suggest you provide your local test results

I tested the BNB quantization locally. The model can be loaded, and the generated results look reasonable. Here are the results I generated using both vLLM and HF:

vLLM:

WARNING 11-14 12:55:32 config.py:440] bitsandbytes quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 11-14 12:55:32 llm_engine.py:249] Initializing an LLM engine (v0.1.dev3385+g3a28f18.d20241112) with config: model='/mnt/Models/llm_models/BaseModel/idefics/Idefics3-8B-Llama3', speculative_config=None, tokenizer='/mnt/Models/llm_models/BaseModel/idefics/Idefics3-8B-Llama3', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.BITSANDBYTES, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=bitsandbytes, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/mnt/Models/llm_models/BaseModel/idefics/Idefics3-8B-Llama3, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, chat_template_text_format=string, mm_processor_kwargs=None, pooler_config=None)
INFO 11-14 12:55:32 selector.py:135] Using Flash Attention backend.
INFO 11-14 12:55:33 model_runner.py:1072] Starting to load model /mnt/Models/llm_models/BaseModel/idefics/Idefics3-8B-Llama3...
INFO 11-14 12:55:34 loader.py:957] Loading weights with BitsAndBytes quantization.  May take a while ...
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:01,  2.59it/s]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.43it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.20it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.02it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.14it/s]

INFO 11-14 12:55:37 model_runner.py:1077] Loading model weights took 5.5544 GB
INFO 11-14 12:55:40 worker.py:232] Memory profiling results: total_gpu_memory=79.32GiB initial_memory_usage=6.07GiB peak_torch_memory=6.71GiB memory_usage_post_profile=6.09Gib non_torch_memory=0.53GiB kv_cache_size=64.15GiB gpu_memory_utilization=0.90
INFO 11-14 12:55:40 gpu_executor.py:113] # GPU blocks: 32846, # CPU blocks: 2048
INFO 11-14 12:55:40 gpu_executor.py:117] Maximum concurrency for 8192 tokens per request: 64.15x
INFO 11-14 12:55:41 model_runner.py:1400] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 11-14 12:55:41 model_runner.py:1404] If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 11-14 12:55:42 model_runner.py:1518] Graph capturing finished in 0 secs, took 0.05 GiB
INFO 11-14 12:55:42 preprocess.py:215] Your model uses the legacy input pipeline instead of the new multi-modal processor. Please note that the legacy pipeline will be removed in a future release. For more details, see: https://github.com/vllm-project/vllm/issues/10114
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.48it/s, est. speed input: 3459.30 toks/s, output: 17.81 toks/s]
 In this image we can see the dried grass.

HF:

Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00,  2.15it/s]
 In this image we can see the dry grass.

[DarkLight1337]

Looks good, let's merge this!

[B-201]

It seems there are some network issues in the CI. Could you help take a look? @DarkLight1337