Automatically increased max_num_batched_tokens under Mamba align mode

[flutist]

Automatically increased max_num_batched_tokens to accommodate Mamba align mode block_size
solve #36697
When using Mamba cache in align mode, block_size may exceed max_num_batched_tokens, causing alignment issues. This PR automatically bumps max_num_batched_tokens to match block_size when this condition is detected, and emits a warning to notify the user of the change.

Purpose

solve
Assertion failed, In Mamba cache align mode, block_size (2096) must be <= max_num_batched_tokens (2048). [type=assertion_error, input_value=ArgsKwargs((), {'model_co...transfer_config': None}), input_type=ArgsKwargs]

Test Plan

before hot fix, vllm serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name Qwen3.5-27B-AWQ-4bit --gpu-memory-utilization 0.9 --port 8848 -tp 2 --max-model-len 131072 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-num-seqs 16 --enable-prefix-caching show error

vllm serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name Qwen3.5-27B-AWQ-4bit --gpu-memory-utilization 0.9 --port 8848 -tp 2 --max-model-len 131072 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-num-seqs 16 --enable-prefix-caching
(APIServer pid=140954) INFO 03-17 11:29:48 [utils.py:302] 
(APIServer pid=140954) INFO 03-17 11:29:48 [utils.py:302]        █     █     █▄   ▄█
(APIServer pid=140954) INFO 03-17 11:29:48 [utils.py:302]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.17.1
(APIServer pid=140954) INFO 03-17 11:29:48 [utils.py:302]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.5-122B-A10B-GPTQ-Int4
(APIServer pid=140954) INFO 03-17 11:29:48 [utils.py:302]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=140954) INFO 03-17 11:29:48 [utils.py:302] 
(APIServer pid=140954) INFO 03-17 11:29:48 [utils.py:238] non-default args: {'model_tag': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'port': 8848, 'model': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', 'max_model_len': 131072, 'served_model_name': ['Qwen3.5-27B-AWQ-4bit'], 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'enable_prefix_caching': True, 'max_num_seqs': 16}
config.json: 4.33kB [00:00, 24.6MB/s]
preprocessor_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 390/390 [00:00<00:00, 4.23MB/s]
(APIServer pid=140954) INFO 03-17 11:29:55 [model.py:531] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=140954) INFO 03-17 11:29:55 [model.py:1554] Using max model len 131072
(APIServer pid=140954) INFO 03-17 11:29:55 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
(APIServer pid=140954) INFO 03-17 11:29:55 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=140954) WARNING 03-17 11:29:55 [config.py:381] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=140954) INFO 03-17 11:29:55 [config.py:401] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=140954) INFO 03-17 11:29:56 [config.py:544] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size.
(APIServer pid=140954) INFO 03-17 11:29:56 [config.py:575] Padding mamba page size by 0.58% to ensure that mamba page size and attention page size are exactly equal.
model.safetensors.index.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 18.1M/18.1M [00:02<00:00, 8.21MB/s]
Parse safetensors files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [00:02<00:00, 14.86it/s]
(APIServer pid=140954) INFO 03-17 11:30:02 [vllm.py:747] Asynchronous scheduling is enabled.
(APIServer pid=140954) Traceback (most recent call last):
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/bin/vllm", line 6, in <module>
(APIServer pid=140954)     sys.exit(main())
(APIServer pid=140954)              ^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=140954)     args.dispatch_function(args)
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 112, in cmd
(APIServer pid=140954)     uvloop.run(run_server(args))
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=140954)     return __asyncio.run(
(APIServer pid=140954)            ^^^^^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=140954)     return runner.run(main)
(APIServer pid=140954)            ^^^^^^^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=140954)     return self._loop.run_until_complete(task)
(APIServer pid=140954)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=140954)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=140954)     return await main
(APIServer pid=140954)            ^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 471, in run_server
(APIServer pid=140954)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 490, in run_server_worker
(APIServer pid=140954)     async with build_async_engine_client(
(APIServer pid=140954)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=140954)     return await anext(self.gen)
(APIServer pid=140954)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 96, in build_async_engine_client
(APIServer pid=140954)     async with build_async_engine_client_from_engine_args(
(APIServer pid=140954)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=140954)     return await anext(self.gen)
(APIServer pid=140954)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 122, in build_async_engine_client_from_engine_args
(APIServer pid=140954)     vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=140954)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1890, in create_engine_config
(APIServer pid=140954)     config = VllmConfig(
(APIServer pid=140954)              ^^^^^^^^^^^
(APIServer pid=140954)   File "/home/admin/miniconda3/envs/for_test/lib/python3.12/site-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__
(APIServer pid=140954)     s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=140954) pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig
(APIServer pid=140954)   Assertion failed, In Mamba cache align mode, block_size (2096) must be <= max_num_batched_tokens (2048). [type=assertion_error, input_value=ArgsKwargs((), {'model_co...transfer_config': None}), input_type=ArgsKwargs]
(APIServer pid=140954)     For further information visit https://errors.pydantic.dev/2.12/v/assertion_error

Test Result

After fix, it work.

vllm serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name Qwen3.5-27B-AWQ-4bit --gpu-memory-utilization 0.9 --port 8848 -tp 2 --max-model-len 131072 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-num-seqs 16 --enable-prefix-caching
(APIServer pid=147681) INFO 03-17 11:33:58 [utils.py:297] 
(APIServer pid=147681) INFO 03-17 11:33:58 [utils.py:297]        █     █     █▄   ▄█
(APIServer pid=147681) INFO 03-17 11:33:58 [utils.py:297]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.1.dev14845+g57a314d15.d20260316
(APIServer pid=147681) INFO 03-17 11:33:58 [utils.py:297]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.5-122B-A10B-GPTQ-Int4
(APIServer pid=147681) INFO 03-17 11:33:58 [utils.py:297]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=147681) INFO 03-17 11:33:58 [utils.py:297] 
(APIServer pid=147681) INFO 03-17 11:33:58 [utils.py:233] non-default args: {'model_tag': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'port': 8848, 'model': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', 'max_model_len': 131072, 'served_model_name': ['Qwen3.5-27B-AWQ-4bit'], 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'enable_prefix_caching': True, 'max_num_seqs': 16}
(APIServer pid=147681) INFO 03-17 11:34:10 [model.py:533] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=147681) INFO 03-17 11:34:10 [model.py:1582] Using max model len 131072
(APIServer pid=147681) INFO 03-17 11:34:12 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
(APIServer pid=147681) INFO 03-17 11:34:12 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=147681) WARNING 03-17 11:34:12 [config.py:385] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=147681) INFO 03-17 11:34:12 [config.py:405] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=147681) INFO 03-17 11:34:12 [config.py:212] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size.
(APIServer pid=147681) WARNING 03-17 11:34:12 [config.py:224] Automatically increased max_num_batched_tokens from 2048 to 2096 to accommodate Mamba align mode block_size
(APIServer pid=147681) INFO 03-17 11:34:12 [config.py:256] Padding mamba page size by 0.58% to ensure that mamba page size and attention page size are exactly equal.
Parse safetensors files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [00:01<00:00, 20.07it/s]
(APIServer pid=147681) INFO 03-17 11:34:16 [vllm.py:754] Asynchronous scheduling is enabled.
tokenizer_config.json: 16.7kB [00:00, 92.1MB/s]
vocab.json: 6.72MB [00:00, 143MB/s]
merges.txt: 3.35MB [00:00, 157MB/s]
tokenizer.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.8M/12.8M [00:01<00:00, 10.6MB/s]
chat_template.jinja: 7.76kB [00:00, 46.4MB/s]
generation_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 244/244 [00:00<00:00, 3.33MB/s]
video_preprocessor_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 385/385 [00:00<00:00, 4.40MB/s]
(EngineCore pid=149493) INFO 03-17 11:34:42 [core.py:103] Initializing a V1 LLM engine (v0.1.dev14845+g57a314d15.d20260316) with config: model='Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.5-27B-AWQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=149493) WARNING 03-17 11:34:42 [multiproc_executor.py:997] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=149493) INFO 03-17 11:34:42 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=33.1.35.33 (local), world_size=2, local_world_size=2
(Worker pid=149761) INFO 03-17 11:34:52 [parallel_state.py:1395] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:47983 backend=nccl
(Worker pid=149883) INFO 03-17 11:34:55 [parallel_state.py:1395] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:47983 backend=nccl
(Worker pid=149883) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=149761) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=149883) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=149761) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=149761) INFO 03-17 11:34:56 [pynccl.py:111] vLLM is using nccl==2.27.5
(Worker pid=149761) WARNING 03-17 11:34:56 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available.
(Worker pid=149883) WARNING 03-17 11:34:56 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available.
(Worker pid=149883) INFO 03-17 11:34:56 [parallel_state.py:1717] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1, EPLB rank N/A
(Worker pid=149761) INFO 03-17 11:34:56 [parallel_state.py:1717] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker_TP0 pid=149761) INFO 03-17 11:35:10 [gpu_model_runner.py:4492] Starting to load model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4...
(Worker_TP1 pid=149883) INFO 03-17 11:35:10 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_TP1 pid=149883) INFO 03-17 11:35:10 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP0 pid=149761) INFO 03-17 11:35:10 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_TP0 pid=149761) INFO 03-17 11:35:10 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP1 pid=149883) INFO 03-17 11:35:11 [qwen3_next.py:198] Using Triton/FLA GDN prefill kernel
(Worker_TP0 pid=149761) INFO 03-17 11:35:11 [qwen3_next.py:198] Using Triton/FLA GDN prefill kernel
(Worker_TP0 pid=149761) INFO 03-17 11:35:11 [cuda.py:317] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=149761) INFO 03-17 11:35:11 [flash_attn.py:593] Using FlashAttention version 2
model.safetensors-00005-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:20<00:00, 81.8MB/s]
model.safetensors-00003-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:20<00:00, 81.1MB/s]
model.safetensors-00004-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:20<00:00, 81.1MB/s]
model.safetensors-00002-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:21<00:00, 80.3MB/s]
model.safetensors-00007-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:22<00:00, 76.7MB/s]
model.safetensors-00006-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:26<00:00, 63.4MB/s]
model.safetensors-00001-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:27<00:00, 62.0MB/s]
model.safetensors-00008-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:27<00:00, 61.1MB/s]
model.safetensors-00009-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:19<00:00, 88.7MB/s]
model.safetensors-00013-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:18<00:00, 93.7MB/s]
model.safetensors-00010-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:21<00:00, 80.3MB/s]
model.safetensors-00012-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:22<00:00, 76.0MB/s]
model.safetensors-00011-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:23<00:00, 73.3MB/s]
model.safetensors-00014-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:17<00:00, 98.0MB/s]
model.safetensors-00015-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:18<00:00, 93.7MB/s]
model.safetensors-00016-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:24<00:00, 70.3MB/s]
model.safetensors-00017-of-00039.safeten(…): 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:14<00:00, 115MB/s]
model.safetensors-00018-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:18<00:00, 90.7MB/s]
model.safetensors-00019-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:22<00:00, 76.6MB/s]
model.safetensors-00020-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:22<00:00, 74.6MB/s]
model.safetensors-00022-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:21<00:00, 77.3MB/s]
model.safetensors-00024-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:18<00:00, 93.7MB/s]
model.safetensors-00023-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:25<00:00, 67.5MB/s]
model.safetensors-00021-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.69G/1.69G [00:27<00:00, 60.7MB/s]
model.safetensors-00026-of-00039.safeten(…): 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:12<00:00, 140MB/s]
model.safetensors-00025-of-00039.safeten(…): 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:15<00:00, 106MB/s]
model.safetensors-00027-of-00039.safeten(…): 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:13<00:00, 125MB/s]
model.safetensors-00029-of-00039.safeten(…): 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:16<00:00, 105MB/s]
model.safetensors-00028-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:24<00:00, 68.2MB/s]
model.safetensors-00033-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:24<00:00, 69.9MB/s]
model.safetensors-00031-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:25<00:00, 67.1MB/s]
model.safetensors-00034-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:25<00:00, 65.0MB/s]
model.safetensors-00032-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:27<00:00, 61.2MB/s]
model.safetensors-00030-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:29<00:00, 57.1MB/s]
model.safetensors-00035-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:20<00:00, 80.7MB/s]
model.safetensors-00036-of-00039.safeten(…): 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1.68G/1.68G [00:23<00:00, 72.3MB/s]
model.safetensors-00037-of-00039.safeten(…): 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 6.40G/6.40G [00:26<00:00, 242MB/s]
model.safetensors-00039-of-00039.safeten(…): 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 5.41G/5.41G [00:31<00:00, 172MB/s]
model.safetensors-00038-of-00039.safeten(…): 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 6.44G/6.44G [00:37<00:00, 174MB/s]
(Worker_TP1 pid=149883) INFO 03-17 11:37:49 [weight_utils.py:574] Time spent downloading weights for Qwen/Qwen3.5-122B-A10B-GPTQ-Int4: 152.912391 seconds40G [00:20<00:00, 295MB/s]
Loading safetensors checkpoint shards:   0% Completed | 0/39 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   3% Completed | 1/39 [00:13<08:14, 13.02s/it]
Loading safetensors checkpoint shards:   5% Completed | 2/39 [00:26<08:09, 13.22s/it]
Loading safetensors checkpoint shards:   8% Completed | 3/39 [00:37<07:27, 12.42s/it]
Loading safetensors checkpoint shards:  10% Completed | 4/39 [00:50<07:20, 12.60s/it]
Loading safetensors checkpoint shards:  13% Completed | 5/39 [01:04<07:27, 13.17s/it]
Loading safetensors checkpoint shards:  15% Completed | 6/39 [01:17<07:10, 13.06s/it]
Loading safetensors checkpoint shards:  18% Completed | 7/39 [01:30<06:58, 13.07s/it]
Loading safetensors checkpoint shards:  21% Completed | 8/39 [01:43<06:39, 12.89s/it]
Loading safetensors checkpoint shards:  23% Completed | 9/39 [01:54<06:12, 12.41s/it]
Loading safetensors checkpoint shards:  26% Completed | 10/39 [02:08<06:10, 12.79s/it]
Loading safetensors checkpoint shards:  28% Completed | 11/39 [02:22<06:07, 13.14s/it]
Loading safetensors checkpoint shards:  31% Completed | 12/39 [02:35<05:59, 13.32s/it]
Loading safetensors checkpoint shards:  33% Completed | 13/39 [02:50<05:56, 13.71s/it]
Loading safetensors checkpoint shards:  36% Completed | 14/39 [03:05<05:48, 13.95s/it]
Loading safetensors checkpoint shards:  38% Completed | 15/39 [03:19<05:36, 14.03s/it]
Loading safetensors checkpoint shards:  41% Completed | 16/39 [03:36<05:41, 14.86s/it]
Loading safetensors checkpoint shards:  44% Completed | 17/39 [03:51<05:33, 15.14s/it]
Loading safetensors checkpoint shards:  46% Completed | 18/39 [04:06<05:15, 15.04s/it]
Loading safetensors checkpoint shards:  49% Completed | 19/39 [04:18<04:42, 14.13s/it]
Loading safetensors checkpoint shards:  51% Completed | 20/39 [04:32<04:28, 14.11s/it]
Loading safetensors checkpoint shards:  54% Completed | 21/39 [04:46<04:11, 13.96s/it]
Loading safetensors checkpoint shards:  56% Completed | 22/39 [05:01<04:03, 14.33s/it]
Loading safetensors checkpoint shards:  59% Completed | 23/39 [05:16<03:50, 14.41s/it]
Loading safetensors checkpoint shards:  62% Completed | 24/39 [05:29<03:32, 14.19s/it]
Loading safetensors checkpoint shards:  64% Completed | 25/39 [05:42<03:10, 13.62s/it]
Loading safetensors checkpoint shards:  67% Completed | 26/39 [05:53<02:49, 13.04s/it]
Loading safetensors checkpoint shards:  69% Completed | 27/39 [06:03<02:25, 12.14s/it]
Loading safetensors checkpoint shards:  72% Completed | 28/39 [06:15<02:13, 12.09s/it]
Loading safetensors checkpoint shards:  74% Completed | 29/39 [06:30<02:08, 12.90s/it]
Loading safetensors checkpoint shards:  77% Completed | 30/39 [06:43<01:56, 12.90s/it]
Loading safetensors checkpoint shards:  79% Completed | 31/39 [06:57<01:45, 13.16s/it]
Loading safetensors checkpoint shards:  82% Completed | 32/39 [07:09<01:30, 12.89s/it]
Loading safetensors checkpoint shards:  85% Completed | 33/39 [07:21<01:15, 12.59s/it]
Loading safetensors checkpoint shards:  87% Completed | 34/39 [07:34<01:03, 12.67s/it]
Loading safetensors checkpoint shards:  90% Completed | 35/39 [07:52<00:57, 14.38s/it]
Loading safetensors checkpoint shards:  92% Completed | 36/39 [08:06<00:42, 14.31s/it]
Loading safetensors checkpoint shards:  95% Completed | 37/39 [08:45<00:43, 21.53s/it]
Loading safetensors checkpoint shards:  97% Completed | 38/39 [09:54<00:35, 35.91s/it]
Loading safetensors checkpoint shards: 100% Completed | 39/39 [10:12<00:00, 30.45s/it]
Loading safetensors checkpoint shards: 100% Completed | 39/39 [10:12<00:00, 15.70s/it]
(Worker_TP0 pid=149761) 
(Worker_TP0 pid=149761) INFO 03-17 11:48:02 [default_loader.py:373] Loading weights took 612.40 seconds
(Worker_TP0 pid=149761) INFO 03-17 11:48:04 [gpu_model_runner.py:4577] Model loading took 34.32 GiB memory and 773.225700 seconds
(Worker_TP1 pid=149883) INFO 03-17 11:48:04 [gpu_model_runner.py:5499] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0 pid=149761) INFO 03-17 11:48:04 [gpu_model_runner.py:5499] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0 pid=149761) INFO 03-17 11:48:14 [backends.py:988] Using cache directory: /home/admin/.cache/vllm/torch_compile_cache/77532660c9/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=149761) INFO 03-17 11:48:14 [backends.py:1048] Dynamo bytecode transform time: 7.49 s
(Worker_TP1 pid=149883) INFO 03-17 11:48:18 [backends.py:371] Cache the graph of compile range (1, 2096) for later use
(Worker_TP0 pid=149761) INFO 03-17 11:48:18 [backends.py:371] Cache the graph of compile range (1, 2096) for later use
(Worker_TP0 pid=149761) INFO 03-17 11:48:37 [backends.py:387] Compiling a graph for compile range (1, 2096) takes 22.10 s
(Worker_TP1 pid=149883) /home/admin/workspace/aop_lab/app_source/vllm_custom_dataset_img_support_base64/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (32). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(Worker_TP1 pid=149883)   return fn(*contiguous_args, **contiguous_kwargs)
(Worker_TP0 pid=149761) INFO 03-17 11:48:39 [decorators.py:627] saved AOT compiled function to /home/admin/.cache/vllm/torch_compile_cache/torch_aot_compile/6dcad07d427156b5cff90bf8a938953d4d4cc536dece8ca0f6cc17b416f33296/rank_0_0/model
(Worker_TP0 pid=149761) INFO 03-17 11:48:39 [monitor.py:48] torch.compile took 31.79 s in total
(Worker_TP0 pid=149761) /home/admin/workspace/aop_lab/app_source/vllm_custom_dataset_img_support_base64/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (32). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(Worker_TP0 pid=149761)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=149493) INFO 03-17 11:49:05 [shm_broadcast.py:681] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=149761) /home/admin/workspace/aop_lab/app_source/vllm_custom_dataset_img_support_base64/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (32). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(Worker_TP0 pid=149761)   return fn(*contiguous_args, **contiguous_kwargs)
(Worker_TP1 pid=149883) /home/admin/workspace/aop_lab/app_source/vllm_custom_dataset_img_support_base64/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (32). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(Worker_TP1 pid=149883)   return fn(*contiguous_args, **contiguous_kwargs)
(Worker_TP0 pid=149761) INFO 03-17 11:50:04 [monitor.py:76] Initial profiling/warmup run took 85.05 s
(EngineCore pid=149493) INFO 03-17 11:50:05 [shm_broadcast.py:681] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP1 pid=149883) INFO 03-17 11:50:11 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=32
(Worker_TP0 pid=149761) INFO 03-17 11:50:11 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=32
(Worker_TP1 pid=149883) INFO 03-17 11:50:11 [gpu_model_runner.py:5618] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP0 pid=149761) INFO 03-17 11:50:11 [gpu_model_runner.py:5618] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP0 pid=149761) INFO 03-17 11:50:13 [custom_all_reduce.py:216] Registering 384 cuda graph addresses
(Worker_TP1 pid=149883) INFO 03-17 11:50:13 [custom_all_reduce.py:216] Registering 384 cuda graph addresses
(Worker_TP0 pid=149761) INFO 03-17 11:50:14 [gpu_model_runner.py:5697] Estimated CUDA graph memory: 1.13 GiB total
(Worker_TP1 pid=149883) INFO 03-17 11:50:14 [gpu_model_runner.py:5697] Estimated CUDA graph memory: 1.13 GiB total
(Worker_TP1 pid=149883) INFO 03-17 11:50:15 [gpu_worker.py:486] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9253 to maintain the same effective KV cache size.
(Worker_TP0 pid=149761) INFO 03-17 11:50:15 [gpu_worker.py:452] Available KV cache memory: 3.86 GiB
(Worker_TP0 pid=149761) INFO 03-17 11:50:15 [gpu_worker.py:486] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9253 to maintain the same effective KV cache size.
(EngineCore pid=149493) INFO 03-17 11:50:15 [kv_cache_utils.py:1316] GPU KV cache size: 83,840 tokens
(EngineCore pid=149493) INFO 03-17 11:50:15 [kv_cache_utils.py:1321] Maximum concurrency for 131,072 tokens per request: 2.32x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|███████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:01<00:00,  6.20it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00,  5.21it/s]
(Worker_TP1 pid=149883) INFO 03-17 11:50:17 [custom_all_reduce.py:216] Registering 1152 cuda graph addresses
(Worker_TP0 pid=149761) INFO 03-17 11:50:17 [custom_all_reduce.py:216] Registering 1152 cuda graph addresses
(Worker_TP0 pid=149761) INFO 03-17 11:50:18 [gpu_model_runner.py:5757] Graph capturing finished in 3 secs, took 0.78 GiB
(Worker_TP0 pid=149761) INFO 03-17 11:50:18 [gpu_worker.py:614] CUDA graph pool memory: 0.78 GiB (actual), 1.13 GiB (estimated), difference: 0.35 GiB (45.3%).
(Worker_TP1 pid=149883) INFO 03-17 11:50:18 [gpu_worker.py:614] CUDA graph pool memory: 0.78 GiB (actual), 1.13 GiB (estimated), difference: 0.35 GiB (45.3%).
(EngineCore pid=149493) INFO 03-17 11:50:18 [core.py:281] init engine (profile, create kv cache, warmup model) took 133.80 seconds
(EngineCore pid=149493) INFO 03-17 11:50:33 [vllm.py:754] Asynchronous scheduling is enabled.
(APIServer pid=147681) INFO 03-17 11:50:33 [api_server.py:573] Supported tasks: ['generate']
(APIServer pid=147681) INFO 03-17 11:50:35 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=147681) WARNING 03-17 11:50:35 [model.py:1376] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=147681) INFO 03-17 11:50:35 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=147681) INFO 03-17 11:50:35 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=147681) INFO 03-17 11:50:41 [hf.py:320] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=147681) INFO 03-17 11:50:49 [base.py:216] Multi-modal warmup completed in 7.982s
(APIServer pid=147681) INFO 03-17 11:50:49 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=147681) INFO 03-17 11:50:50 [api_server.py:577] Starting vLLM server on http://0.0.0.0:8848
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:37] Available routes are:
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=147681) INFO 03-17 11:50:50 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=147681) INFO:     Started server process [147681]
(APIServer pid=147681) INFO:     Waiting for application startup.
(APIServer pid=147681) INFO:     Application startup complete.

And completion is work.

# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8848/v1/chat/completions" \
>     -H "Content-Type: application/json" \
>     --data '{
>         "model": "Qwen3.5-27B-AWQ-4bit",
>         "messages": [
>             {
>                 "role": "user",
>                 "content": "What is the capital of France?"
>             }
>         ]
>     }'



{"id":"chatcmpl-9ad82c292b9872ab","object":"chat.completion","created":1773719595,"model":"Qwen3.5-27B-AWQ-4bit","choices":[{"index":0,"message":{"role":"assistant","content":"\n\nThe capital of France is **Paris**.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":"Thinking Process:\n\n1.  **Identify the core question:** The user is asking \"What is the capital of France?\"\n2.  **Retrieve knowledge:** Access general knowledge about geography and countries.\n3.  **Verify the fact:** The capital of France is Paris.\n4.  **Formulate the answer:** State the fact clearly and concisely.\n5.  **Final Output:** \"The capital of France is Paris.\" or simply \"Paris.\" (A complete sentence is usually better for politeness).\n\n*Self-Correction/Refinement:* Keep it direct. No need for extra fluff unless asked.\n\n*Draft:* The capital of France is Paris.\n*Final Polish:* The capital of France is Paris.cw\n"},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":17,"total_tokens":185,"completion_tokens":168,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}(base) 

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a helpful change to automatically increase max_num_batched_tokens to match block_size when Mamba's align cache mode is used, preventing a potential assertion failure. The implementation is sound, but I've found a minor issue in the warning log message where the old and new token values are swapped, which could be misleading. I've provided a suggestion to correct the order of arguments in the log message.

[flutist]

@DarkLight1337 PTAL

[flutist]

@hmellor @mgoin @Isotr0py @noooop @NickLucche @tdoublep @ywang96 @ProExpertProg Sorry to bother you, but could you please help me merge this PR file? This solved the problem. If there's anything else I can do, I'll continue. I'm very happy to hear your response.

[flutist]

@hmellor @mgoin @Isotr0py @noooop @NickLucche @tdoublep @ywang96 @ProExpertProg Sorry to bother you, but could you please help me merge this PR file? This solved the problem. If there's anything else I can do, I'll continue. I'm very happy to hear your response.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @flutist.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork