FEAT: support deepseek-v2 and 2.5 #2292
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
September 12, 2024 11:26
# Conflicts: # xinference/model/llm/llm_family.json # xinference/model/llm/llm_family_modelscope.json # xinference/model/llm/transformers/core.py
ChengjieLi28
approved these changes
Sep 13, 2024
Vanocore
added a commit
to Vanocore/xinference
that referenced
this pull request
Sep 22, 2024
* 'main' of https://github.com/xorbitsai/inference: FEAT: support qwen2.5-coder-instruct and qwen2.5 sglang (xorbitsai#2332) DOC: update models for doc and readme (xorbitsai#2330) BUG: fix stable diffusion from dify tool (xorbitsai#2336) BUG: support old register llm format (xorbitsai#2335) FEAT: Support Qwen 2.5 (xorbitsai#2325) BUG: Fix CosyVoice missing output (xorbitsai#2320) BUG: [UI] Fix registration page bug. (xorbitsai#2315) Bug: modify vllm image version (xorbitsai#2312) BUG: modify vllm image version (xorbitsai#2311) FEAT: qwen2 audio (xorbitsai#2271) BUG: fix sampler_name for img2img (xorbitsai#2301) FEAT: Support yi-coder-chat (xorbitsai#2302) FEAT: support flux.1 image2image and inpainting (xorbitsai#2296) FEAT: support sdapi/img2img (xorbitsai#2293) ENH: Support fish speech 1.4 (xorbitsai#2295) FEAT: Update Qwen2-VL-Model to support flash_attention_2 implementation (xorbitsai#2289) FEAT: support deepseek-v2 and 2.5 (xorbitsai#2292) # Conflicts: # xinference/model/audio/cosyvoice.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
测试环境： NVIDIA GeForce RTX 3090 Ti * 2
测试模型：DeepSeek-V2-Lite-Chat
推理引擎：vllm==0.5.5
测试参数：--num-prompt 100 --concurrency 50
推理引擎：transformers==4.39.3
测试参数：--num-prompt 50 --concurrency 5
推理引擎：sglang
模型参数：mem_fraction_static=0.8
测试参数：--num-prompt 100 --concurrency 50