Skip to content
This repository was archived by the owner on Sep 4, 2025. It is now read-only.

[do-not-merge] Ibm 20241105 #218

Closed
wants to merge 278 commits into from
Closed

[do-not-merge] Ibm 20241105 #218

wants to merge 278 commits into from

Conversation

@fialhocoelho
Copy link

@fialhocoelho fialhocoelho commented Nov 5, 2024

Image build.

DarkLight1337 and others added 30 commits October 17, 2024 13:55
@DarkLight1337
[Misc] Print stack trace using logger.exception (vllm-project#9461)
390be74
@LucasWilkinson @mgoin
[misc] CUDA Time Layerwise Profiler (vllm-project#8337)
9d30a05 
Co-authored-by: Varun Sundar Rabindranath <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
@sasha0552
[Bugfix] Allow prefill of assistant response when using `mistral_comm…
5e443b5 
…on` (vllm-project#9446)
@WoosukKwon
[TPU] Call torch._sync(param) during weight loading (vllm-project#9437)
8e1cddc
@bigPYJ1151
[Hardware][CPU] compressed-tensor INT8 W8A8 AZP support (vllm-project…
5eda21e 
…#9344)
@KuntaiDu
[Core] Deprecating block manager v1 and make block manager v2 default (…
81ede99 
…vllm-project#8704)

Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
@dtrifiro
[CI/Build] remove .github from .dockerignore, add dirty repo check (v…
a2c71c5 
…llm-project#9375)
@DarkLight1337
[Misc] Remove commit id file (vllm-project#9470)
7871659
@ProExpertProg
[torch.compile] Fine-grained CustomOp enabling mechanism (vllm-projec…
0f41fbe 
…t#9300)
@bnellnm
[Bugfix] Fix support for dimension like integers and ScalarType (vllm…
eca2c5f 
…-project#9299)
@wukaixingxp @Isotr0py
[Bugfix] Add random_seed to sample_hf_requests in benchmark_serving s…
d65049d 
…cript (vllm-project#9013)

Co-authored-by: Isotr0py <[email protected]>
@sasha0552
[Bugfix] Print warnings related to mistral_common tokenizer only on…
d615b5c 
…ce (vllm-project#9468)
@sssrijan-amazon
[Hardwware][Neuron] Simplify model load for transformers-neuronx libr…
bb76538 
…ary (vllm-project#9380)
@robertgshaw2-redhat @afeldman-nm
@WoosukKwon @laishzh @maxdebayser @DarkLight1337
Support BERTModel (first encoder-only embedding model) (vllm-proj…
343f8e0 
…ect#9056)

Signed-off-by: Max de Bayser <[email protected]>
Signed-off-by: Max de Bayser <[email protected]>
Co-authored-by: Andrew Feldman <[email protected]>
Co-authored-by: afeldman-nm <[email protected]>
Co-authored-by: Woosuk Kwon <[email protected]>
Co-authored-by: laishzh <[email protected]>
Co-authored-by: Max de Bayser <[email protected]>
Co-authored-by: Max de Bayser <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>
@dsikka
[BugFix] Stop silent failures on compressed-tensors parsing (vllm-pro…
48138a8 
…ject#9381)
@joerunde
[Bugfix][Core] Use torch.cuda.memory_stats() to profile peak memory u…
de4008e 
…sage (vllm-project#9352)

Signed-off-by: Joe Runde <[email protected]>
@blueyo0
[Qwen2.5] Support bnb quant for Qwen2.5 (vllm-project#9467)
154a8ae
@russellb
[CI/Build] Use commit hash references for github actions (vllm-projec…
944dd8e 
…t#9430)
@njhill
[BugFix] Typing fixes to RequestOutput.prompt and beam search (vllm-p…
1ffc8a7 
…roject#9473)
@tomeras91
[Frontend][Feature] Add jamba tool parser (vllm-project#9154)
d2b1bf5
@njhill
[BugFix] Fix and simplify completion API usage streaming (vllm-projec…
25aeb7d 
…t#9475)
@DarkLight1337
[CI/Build] Fix lint errors in mistral tokenizer (vllm-project#9504)
1bbbcc0
@tlrmchlsmth
[Bugfix] Fix offline_inference_with_prefix.py (vllm-project#9505)
ae8b633
@russellb
[Misc] benchmark: Add option to set max concurrency (vllm-project#9390)
7dbe738 
Signed-off-by: Russell Bryant <[email protected]>
@DarkLight1337
[Model] Add user-configurable task for models that support both gener…
051eaf6 
…ation and embedding (vllm-project#9424)
@russellb
[CI/Build] Add error matching config for mypy (vllm-project#9512)
67a7e5e
@mgoin
[Model] Support Pixtral models in the HF Transformers format (vllm-pr…
3921a2f 
…oject#9036)
@coolkp
[MISC] Add lora requests to metrics (vllm-project#9477)
9bb10a7 
Co-authored-by: Kunjan Patel <kunjanp_google_com@vllm.us-central1-a.c.kunjanp-gke-dev-2.internal>
@comaniac
[MISC] Consolidate cleanup() and refactor offline_inference_with_pref…
d11bf43 
…ix.py (vllm-project#9510)
@tdoublep @cyang49 @comaniac
[Kernel] Add env variable to force flashinfer backend to enable tenso…
0c9a525 
…r cores (vllm-project#9497)

Signed-off-by: Thomas Parnell <[email protected]>
Co-authored-by: Chih-Chieh Yang <[email protected]>
Co-authored-by: Cody Yu <[email protected]>
robertgshaw2-redhat and others added 15 commits November 4, 2024 16:01
@robertgshaw2-redhat
[Bugfix] Fix MQLLMEngine hanging (vllm-project#9973)
04cef2c 
Signed-off-by: [email protected] <[email protected]>
@lk-chen
[Misc] Refactor benchmark_throughput.py (vllm-project#9779)
9a5664d 
Signed-off-by: Linkun Chen <[email protected]>
Co-authored-by: Linkun Chen <[email protected]>
Co-authored-by: Linkun Chen <[email protected]>
@tomeras91
[Frontend] Add max_tokens prometheus metric (vllm-project#9881)
ac04a97 
Signed-off-by: Tomer Asida <[email protected]>
@bnellnm
[Bugfix] Upgrade to pytorch 2.5.1 (vllm-project#10001)
d93478b 
Signed-off-by: Bill Nell <[email protected]>
@youkaichao
[4.5/N] bugfix for quant config in speculative decode (vllm-project#1…
2094062 
…0007)

Signed-off-by: youkaichao <[email protected]>
@mgoin
[Bugfix] Respect modules_to_not_convert within awq_marlin (vllm-proje…
8f0a9ca 
…ct#9895)

Signed-off-by: mgoin <[email protected]>
@tlrmchlsmth
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (vll…
04bbf38 
…m-project#9994)

Signed-off-by: Tyler Michael Smith <[email protected]>
@DarkLight1337
[Core] Make encoder-decoder inputs a nested structure to be more comp…
bbc3619 
…osable (vllm-project#9604)

Signed-off-by: DarkLight1337 <[email protected]>
@tlrmchlsmth
[Bugfix] Fixup Mamba (vllm-project#10004)
ad23318 
Signed-off-by: Tyler Michael Smith <[email protected]>
@GeneDer
[BugFix] Lazy import ray (vllm-project#10021)
7a83b1a
@chaunceyjiang
[Misc] vllm CLI flags should be ordered for better user readability (v…
93dee88 
…llm-project#10017)

Signed-off-by: chaunceyjiang <[email protected]>
@fialhocoelho
Merge branch 'main' of https://github.com/vllm-project/vllm into ibm-…
476349a 
…20241105

Signed-off-by: Jefferson Fialho <[email protected]>
@fialhocoelho
Squash 5733
3895560 
Signed-off-by: Jefferson Fialho <[email protected]>
@fialhocoelho
Squash 6357
04c478d 
Signed-off-by: Jefferson Fialho <[email protected]>
@fialhocoelho
Squash 9027
5f01cf0 
Signed-off-by: Jefferson Fialho <[email protected]>
@fialhocoelho fialhocoelho self-assigned this Nov 5, 2024
@fialhocoelho fialhocoelho requested a review from njhill as a code owner November 5, 2024 15:14
@openshift-ci openshift-ci bot requested review from heyselbi and rpancham November 5, 2024 15:14
@openshift-ci
Copy link

openshift-ci bot commented Nov 5, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: fialhocoelho
Once this PR has been reviewed and has the lgtm label, please assign danielezonca for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fialhocoelho
Squash 217
d314f06 
Signed-off-by: Jefferson Fialho <[email protected]>
@fialhocoelho
Solving dependencies incompatibilities
28b5bdf 
Signed-off-by: Jefferson Fialho <[email protected]>

Pin tiktoken >=0.7.0, <0.8.0

Signed-off-by: Jefferson Fialho <[email protected]>

Pin tiktoken==0.7.0

Signed-off-by: Jefferson Fialho <[email protected]>

Pin pillow==10.4.0

Signed-off-by: Jefferson Fialho <[email protected]>

pin pytorch in cmake list

Signed-off-by: Jefferson Fialho <[email protected]>
@fialhocoelho
Squash 9522
25c55e4 
Signed-off-by: Jefferson Fialho <[email protected]>
@fialhocoelho
Install adapter merged with PR 172
9b51d3d 
Signed-off-by: Jefferson Fialho <[email protected]>
@openshift-ci
Copy link

openshift-ci bot commented Nov 6, 2024

@fialhocoelho: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/rocm-pr-image-mirror 9b51d3d link true /test rocm-pr-image-mirror
ci/prow/images 9b51d3d link true /test images
ci/prow/smoke-test 9b51d3d link true /test smoke-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

groenenboomj pushed a commit that referenced this pull request Feb 24, 2025
@AdrianAbeyta
Improve profiling setup and documentation, sync benchmarks with main (#…
4cb422f 
…218)

* Automatically set rpd env var with profile flag

* Add readme

* Fix lint errors

---------

Co-authored-by: AdrianAbeyta <[email protected]>
dtrifiro pushed a commit to red-hat-data-services/vllm that referenced this pull request May 13, 2025
@andy-neuma
Sync midstream to upstream v0.8.5.post1 (opendatahub-io#218)
7db80a0 
Syncing midstream NM fork to Upstream tag of
[v0.8.5.post1](https://github.com/vllm-project/vllm/tree/v0.8.5.post1) +
cherry pick of
vllm-project@be633fb
needed for benchmarks +
[CP](neuralmagic/nm-vllm-ent@1fe447d)
for compressed tensor bump +
[CP](vllm-project#17677) for lora on AMD +
[CP](vllm-project#17315) for llama4 w/ pure
dense layers

```
commit 31c73ba (HEAD -> upstream-v0.8.5, nm-fork/upstream-v0.8.5)
Author: Chauncey <[email protected]>
Date:   Wed Apr 30 15:11:04 2025 +0800

    [Bugfix] Fix AttributeError: 'State' object has no attribute 'engine_client' (vllm-project#17434)
    
    Signed-off-by: chaunceyjiang <[email protected]>

commit f8db0bd
Author: Lucas Wilkinson <[email protected]>
Date:   Fri May 2 14:01:38 2025 -0400

    [BugFix][Attention] Fix sliding window attention in V1 giving incorrect results (vllm-project#17574)
    
    Signed-off-by: Lucas Wilkinson <[email protected]>

commit e335c34
Author: Robert Shaw <[email protected]>
Date:   Fri May 2 04:07:03 2025 -0400

    [BugFix] Fix Memory Leak (vllm-project#17567)
    
    Signed-off-by: [email protected] <[email protected]>

commit cc463fe
Merge: 1e358ff ba41cc9
Author: Selbi Nuryyeva <[email protected]>
Date:   Tue Apr 29 12:34:57 2025 -0400

    Merge branch 'tag-upstream-v0.8.5' into upstream-v0.8.5

commit ba41cc9 (tag: v0.8.5, tag-upstream-v0.8.5)
Author: Michael Goin <[email protected]>
Date:   Mon Apr 28 16:20:24 2025 -0600

    [Model] Add tuned triton fused_moe configs for Qwen3Moe (vllm-project#17328)
    
    Signed-off-by: mgoin <[email protected]>

commit dcbac4c
Author: Simon Mo <[email protected]>
Date:   Mon Apr 28 14:12:01 2025 -0700

    [Model] Qwen3 Dense FP8 Compat Fixes (vllm-project#17318)
    
    Signed-off-by: simon-mo <[email protected]>
[...]
```

Commands
```
git fetch upstream
git checkout -b upstream-v0.8.5
git merge upstream/v0.8.5
git cherry-pick be633fb
```

TEST PLAN
accept sync:
https://github.com/neuralmagic/nm-cicd/actions/runs/14841223552
related PR in cicd: neuralmagic/nm-cicd#99
release workflow:
https://github.com/neuralmagic/nm-cicd/actions/runs/14845693864
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@heyselbi heyselbi Awaiting requested review from heyselbi

@rpancham rpancham Awaiting requested review from rpancham

Assignees

@fialhocoelho fialhocoelho

Labels

do-not-merge/hold

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.