docker最新版本(1.8.0)加载自定义llm模型不成功 #4050
Description
System Info / 系統信息
Cuda：Version: 12.4
xllamacpp：0.1.23
vllm 0.8.5,
Python 3.10.14,
操作系统 ： Ubuntu 20.04.6 LTS(window的wsl)
镜像版本:xprobe/xinference:latest
Docker version 25.0.3
Docker Compose version v2.24.6-desktop.1
Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？
- docker / docker
- pip install / 通过 pip install 安装
- installation from source / 从源码安装
Version info / 版本信息
LLM模型是基于 DeepSeek-R1-Distill-Qwen-7B 训练的模型
{
"version": 2,
"context_length": 2048,
"model_name": "custom-llm",
"model_lang": [
"en",
"zh"
],
"model_ability": [
"generate"
],
"model_description": "This is a custom model description.",
"model_family": "qwen2.5",
"model_specs": [
{
"model_format": "ggufv2",
"model_size_in_billions": 7,
"quantization": "q4_k_m",
"multimodal_projectors": null,
"model_id": null,
"model_file_name_template": "toone-model-7b-q4_k_m.gguf",
"model_file_name_split_template": null,
"quantization_parts": null,
"model_hub": "huggingface",
"model_uri": "/opt/xinference/model",
"model_revision": null,
"activated_size_in_billions": null
}
],
"chat_template": null,
"stop_token_ids": null,
"stop": null,
"reasoning_start_tag": null,
"reasoning_end_tag": null,
"cache_config": null,
"virtualenv": {
"packages": [],
"inherit_pip_config": true,
"index_url": null,
"extra_index_url": null,
"find_links": null,
"trusted_host": null,
"no_build_isolation": null
}
}The command used to start Xinference / 用以启动 xinference 的命令
启动参数（"Copy as Command Line Command" 操作的值）
xinference launch --model-name custom-llm --model-type LLM --model-engine llama.cpp --model-format ggufv2 --size-in-billions 7 --quantization q4_k_m --n-gpu auto --replica 1 --n-worker 1Reproduction / 复现过程
启动日日志
2025-09-09 19:52:25,514 xinference.core.worker 144 INFO [request 2e7e32d2-8df1-11f0-a05d-0242c0a88002] Enter launch_builtin_model, args: <xinference.core.worker.WorkerActor object at 0x7e10ac3024d0>, kwargs: model_uid=custom-llm-0,model_name=custom-llm,model_size_in_billions=7,model_format=ggufv2,quantization=q4_k_m,model_engine=llama.cpp,model_type=LLM,n_gpu=auto,request_limits=None,peft_model_config=None,gpu_idx=None,download_hub=None,model_path=None,xavier_config=None
2025-09-09 19:52:27,086 xinference.model.llm.cache_manager 144 INFO Cache /opt/xinference/cache/v2/custom-llm-ggufv2-7b-q4_k_m exists
INFO 09-09 19:52:36 [__init__.py:239] Automatically detected platform cuda.
2025-09-09 19:52:44,816 xinference.core.model 2503 INFO Start requests handler.
2025-09-09 19:52:47,016 xinference.model.llm.llama_cpp.core 2503 INFO Try to estimate num gpu layers, n_ctx: 2048, n_batch: 2048, n_parallel: 8, gpus:
[{'caps': {'async': True,
'buffer_from_host_ptr': False,
'events': True,
'host_buffer': True},
'description': 'NVIDIA RTX A6000',
'memory_free': 49854545920,
'memory_total': 51526500352,
'name': 'CUDA0',
'type': <ggml_backend_dev_type.GGML_BACKEND_DEVICE_TYPE_GPU: 1>}]
2025-09-09 19:53:01,634 xinference.model.llm.llama_cpp.core 2503 INFO Estimate num gpu layers: MemoryEstimate(layers=29, graph=1275068416, vram_size=6759806976, total_size=6759806976, tensor_split='', gpu_sizes=[6759806976])
build: 5835 (6491d6e4) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
system info: n_threads = 72, n_threads_batch = 72, total_threads = 72
system_info: n_threads = 72 (n_threads_batch = 72) / 72 | CUDA : ARCHS = 500,610,700,750,800,860,890 | FORCE_MMQ = 1 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
init: loading model
srv load_model: loading model '/opt/xinference/cache/v2/custom-llm-ggufv2-7b-q4_k_m/toone-model-7b-q4_k_m.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA RTX A6000) - 47545 MiB free
llama_model_loader: loaded meta data with 25 key-value pairs and 339 tensors from /opt/xinference/cache/v2/custom-llm-ggufv2-7b-q4_k_m/toone-model-7b-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Smart_Highway_Cost_Deepseek7B_Model
llama_model_loader: - kv 3: general.size_label str = 7.6B
llama_model_loader: - kv 4: qwen2.block_count u32 = 28
llama_model_loader: - kv 5: qwen2.context_length u32 = 131072
llama_model_loader: - kv 6: qwen2.embedding_length u32 = 3584
llama_model_loader: - kv 7: qwen2.feed_forward_length u32 = 18944
llama_model_loader: - kv 8: qwen2.attention.head_count u32 = 28
llama_model_loader: - kv 9: qwen2.attention.head_count_kv u32 = 4
llama_model_loader: - kv 10: qwen2.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 12: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 13: tokenizer.ggml.pre str = deepseek-r1-qwen
llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 16: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 151646
llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 151643
llama_model_loader: - kv 19: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 21: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 22: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 23: general.quantization_version u32 = 2
llama_model_loader: - kv 24: general.file_type u32 = 15
llama_model_loader: - type f32: 141 tensors
llama_model_loader: - type q4_K: 169 tensors
llama_model_loader: - type q6_K: 29 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 4.36 GiB (4.91 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 22
load: token to piece cache size = 0.9310 MB
print_info: arch = qwen2
print_info: vocab_only = 0
print_info: n_ctx_train = 131072
print_info: n_embd = 3584
print_info: n_layer = 28
print_info: n_head = 28
print_info: n_head_kv = 4
print_info: n_rot = 128
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 7
print_info: n_embd_k_gqa = 512
print_info: n_embd_v_gqa = 512
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 18944
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = -1
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 131072
print_info: rope_finetuned = unknown
print_info: model type = 7B
print_info: model params = 7.62 B
print_info: general.name = Smart_Highway_Cost_Deepseek7B_Model
print_info: vocab type = BPE
print_info: n_vocab = 152064
print_info: n_merges = 151387
print_info: BOS token = 151646 '<｜begin▁of▁sentence｜>'
print_info: EOS token = 151643 '<｜end▁of▁sentence｜>'
print_info: EOT token = 151643 '<｜end▁of▁sentence｜>'
print_info: PAD token = 151643 '<｜end▁of▁sentence｜>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<｜end▁of▁sentence｜>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 28 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 29/29 layers to GPU
load_tensors: CPU model buffer size = 292.36 MiB
load_tensors: CUDA0 model buffer size = 4168.09 MiB
....................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max = 8
llama_context: n_ctx = 2048
llama_context: n_ctx_per_seq = 256
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = 0
llama_context: freq_base = 10000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (256) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 4.64 MiB
llama_kv_cache_unified: CUDA0 KV buffer size = 112.00 MiB
llama_kv_cache_unified: size = 112.00 MiB ( 2048 cells, 28 layers, 8 seqs), K (f16): 56.00 MiB, V (f16): 56.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context: CUDA0 compute buffer size = 304.00 MiB
llama_context: CUDA_Host compute buffer size = 11.01 MiB
llama_context: graph nodes = 1098
llama_context: graph splits = 2
common_init_from_params: setting dry_penalty_last_n to ctx_size = 2048
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
ggml_cuda_compute_forward: RMS_NORM failed
CUDA error: device kernel image is invalid
current device: 0, in function ggml_cuda_compute_forward at /home/runner/work/xllamacpp/xllamacpp/thirdparty/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2475
err
/home/runner/work/xllamacpp/xllamacpp/thirdparty/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:78: CUDA error
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(+0x12732d8b)[0x719009f06d8b]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(ggml_print_backtrace+0x21f)[0x719009f071ef]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(ggml_abort+0x152)[0x719009f073c2]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(+0x128a3a36)[0x71900a077a36]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(+0x128b1ad9)[0x71900a085ad9]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(ggml_backend_sched_graph_compute_async+0x41d)[0x719009f1f65d]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0x99)[0x719009e25ca9]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x103)[0x719009e25f63]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(_ZN13llama_context6decodeERK11llama_batch+0x338)[0x719009e2a768]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(llama_decode+0x10)[0x719009e2b940]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(_Z23common_init_from_paramsR13common_params+0x6a5)[0x719009dd5745]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(_ZN14server_context10load_modelERK13common_params+0xdd3)[0x719009cad153]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(+0x124a2828)[0x719009c76828]
/usr/local/lib/python3.10/dist-packages/xllamacpp/xllamacpp.cpython-310-x86_64-linux-gnu.so(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvR13common_paramsR14server_contextSt7promiseIiEESt17reference_wrapperIS3_ESB_IS5_ES8_EEEEE6_M_runEv+0x64)[0x719009c96324]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc2c3)[0x7192d14872c3]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43)[0x7192d224eb43]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x126a00)[0x7192d22e0a00]
2025-09-09 19:53:16,423 xinference.core.worker 144 ERROR Failed to load model custom-llm-0
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/xinference/core/worker.py", line 1117, in launch_builtin_model
await model_ref.load()
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 261, in send
result = await self._wait(future, actor_ref.address, send_message) # type: ignore
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 124, in _wait
return await future
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 104, in _listen
raise ServerClosed(
xoscar.errors.ServerClosed: Remote server unixsocket:///2415919104 closed: 0 bytes read on a total of 11 expected bytes
2025-09-09 19:53:18,815 xinference.core.worker 144 ERROR [request 2e7e32d2-8df1-11f0-a05d-0242c0a88002] Leave launch_builtin_model, error: Remote server unixsocket:///2415919104 closed: 0 bytes read on a total of 11 expected bytes, elapsed time: 53 s
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/xinference/core/utils.py", line 93, in wrapped
ret = await func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/xinference/core/worker.py", line 1117, in launch_builtin_model
await model_ref.load()
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 261, in send
result = await self._wait(future, actor_ref.address, send_message) # type: ignore
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 124, in _wait
return await future
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 104, in _listen
raise ServerClosed(
xoscar.errors.ServerClosed: Remote server unixsocket:///2415919104 closed: 0 bytes read on a total of 11 expected bytes
2025-09-09 19:53:18,825 xinference.api.restful_api 1 ERROR [address=0.0.0.0:38367, pid=144] Remote server unixsocket:///2415919104 closed: 0 bytes read on a total of 11 expected bytes
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/xinference/api/restful_api.py", line 1077, in launch_model
model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/usr/local/lib/python3.10/dist-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
File "/usr/local/lib/python3.10/dist-packages/xinference/core/supervisor.py", line 1244, in launch_builtin_model
await _launch_model()
File "/usr/local/lib/python3.10/dist-packages/xinference/core/supervisor.py", line 1179, in _launch_model
subpool_address = await _launch_one_model(
File "/usr/local/lib/python3.10/dist-packages/xinference/core/supervisor.py", line 1133, in _launch_one_model
subpool_address = await worker_ref.launch_builtin_model(
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/usr/local/lib/python3.10/dist-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
File "/usr/local/lib/python3.10/dist-packages/xinference/core/utils.py", line 93, in wrapped
ret = await func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/xinference/core/worker.py", line 1117, in launch_builtin_model
await model_ref.load()
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 261, in send
result = await self._wait(future, actor_ref.address, send_message) # type: ignore
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 124, in _wait
return await future
File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 104, in _listen
raise ServerClosed(
xoscar.errors.ServerClosed: [address=0.0.0.0:38367, pid=144] Remote server unixsocket:///2415919104 closed: 0 bytes read on a total of 11 expected bytes
Expected behavior / 期待表现
LLM模型是基于 DeepSeek-R1-Distill-Qwen-7B 训练的模型
升级尝试：
尝试升级了(在docker容器内直接升级) xllamacpp 到 0.2.0 ，部署成功了，但调用模型又提示参数需要2个，但去提交了3个(大概这个意思)
xinference 继续升级到1.9.1 还是不行。
====
替换模型：
升级尝试失败后，就考虑文档说明了 DeepSeek-R1-Distill-Qwen-7B 的部署方式。
就重做新环境，离线下载了 DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf 放在 xxx/unsloth/目录下，加载模型正常。
curl 调用 "deepseek-r1-distill-qwen" 可以正常回复内容。
验证通用模型正常后，再重做新环境，把自定义的模型改名为 DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf，按通用模型的方式操作，一切正常，curl调也正常返回。但是 返回的结果不是训练后的结果，是通用的结果。
说明自定义的模型是失败的。另外通用模型的上下文长度是context_length = 131072，自定义的模型只需要2048就够了，不知道怎么配置？
这个模型在ollama的docker环境是正常，并且上下文长度也配置了2048
不知道xinference怎么部署才能得到预期的效果？