This repository was archived by the owner on Oct 11, 2024. It is now read-only.

Upstream sync 2024 06 08#288

Merged

andy-neuma merged 101 commits intomainfrom

upstream-sync-2024-06-08

Jun 10, 2024

Commits on Jun 8, 2024

[Kernel] Add marlin_24 unit tests (#4901)

alexm-redhat
authored and
Robert Shaw
committed
[Kernel] Add flash-attn back (#4907)

WoosukKwon
authored and
Robert Shaw
committed
[Model] LLaVA model refactor (#4910)

DarkLight1337
authored and
Robert Shaw
committed
Remove marlin warning (#4918)

alexm-redhat
authored and
Robert Shaw
committed
[Misc]: allow user to specify port in distributed setting (#4914)

ZwwWayne
authored and
Robert Shaw
committed
[Build/CI] Enabling AMD Entrypoints Test (#4834)

authored and
Robert Shaw
committed
[Bugfix] Fix dummy weight for fp8 (#4916)

authored and
Robert Shaw
committed
[Core] Sharded State Loader download from HF (#4889)

aurickq
authored and
Robert Shaw
committed
[Doc]Add documentation to benchmarking script when running TGI (#4920)

KuntaiDu
authored and
Robert Shaw
committed
[Core] Fix scheduler considering "no LoRA" as "LoRA" (#4897)

Yard1
authored and
Robert Shaw
committed
[Model] add rope_scaling support for qwen2 (#4930)

hzhwcmhf
authored and
Robert Shaw
committed
[Model] Add Phi-2 LoRA support (#4886)

Isotr0py
authored and
Robert Shaw
committed
[Docs] Add acknowledgment for sponsors (#4925)

simon-mo
authored and
Robert Shaw
committed
[CI/Build] Codespell ignore `build/` directory (#4945)

mgoin
authored and
Robert Shaw
committed
[Bugfix] Fix flag name for `max_seq_len_to_capture` (#4935)

kerthcet
authored and
Robert Shaw
committed
[Bugfix][Kernel] Add head size check for attention backend selection (#4944)

Isotr0py
authored and
Robert Shaw
committed
[Frontend] Dynamic RoPE scaling (#4638)

sasha0552
authored and
Robert Shaw
committed
[CI/Build] Enforce style for C++ and CUDA code with `clang-format` (#4722)

mgoin
authored and
Robert Shaw
committed
[misc] remove comments that were supposed to be removed (#4977)

rkooo567
authored and
Robert Shaw
committed
[Kernel] Fixup for CUTLASS kernels in CUDA graphs (#4954)

tlrmchlsmth
authored and
Robert Shaw
committed
[Misc] Load FP8 kv-cache scaling factors from checkpoints (#4893)

comaniac
authored and
Robert Shaw
committed
[Model] LoRA gptbigcode implementation (#3949)

raywanb
authored and
Robert Shaw
committed
[Core] Eliminate parallel worker per-step task scheduling overhead (#4894)

njhill
authored and
Robert Shaw
committed
[Minor] Fix small typo in llama.py: QKVParallelLinear -> QuantizationConfig (#4991)

pcmoritz
authored and
Robert Shaw
committed
[Misc] Take user preference in attention selector (#4960)

comaniac
authored and
Robert Shaw
committed
Marlin 24 prefill performance improvement (about 25% better on average) (#4983)

alexm-redhat
authored and
Robert Shaw
committed
[Bugfix] Update Dockerfile.cpu to fix NameError: name 'vllm_ops' is not defined (#5009)

LetianLee
authored and
Robert Shaw
committed
[Core][1/N] Support send/recv in PyNCCL Groups (#4988)

andoorve
authored and
Robert Shaw
committed
[Kernel] Initial Activation Quantization Support (#4525)

authored and
Robert Shaw
committed
[Core]: Option To Use Prompt Token Ids Inside Logits Processor (#4985)

kezouke
authored and
Robert Shaw
committed
[Doc] add ccache guide in doc (#5012)

authored and
Robert Shaw
committed
[Kernel] Initial Activation Quantization Support (#4525)

authored and
Robert Shaw
committed
[Core][Bugfix]: fix prefix caching for blockv2 (#4764)

authored and
Robert Shaw
committed
[Kernel][Backend][Model] Blocksparse flash attention kernel and Phi-3-Small model (#4799)

authored and
Robert Shaw
committed
[Misc] add logging level env var (#5045)

youkaichao
authored and
Robert Shaw
committed
[Dynamic Spec Decoding] Minor fix for disabling speculative decoding (#5000)

LiuXiaoxuanPKU
authored and
Robert Shaw
committed
[Misc] Make Serving Benchmark More User-friendly (#5044)

ywang96
authored and
Robert Shaw
committed
[Bugfix / Core] Prefix Caching Guards (merged with main) (#4846)

committed
[Core] Allow AQLM on Pascal (#5058)

sasha0552
authored and
Robert Shaw
committed
[Model] Add support for falcon-11B (#5069)

Isotr0py
authored and
Robert Shaw
committed
[Core] Sliding window for block manager v2 (#4545)

authored and
Robert Shaw
committed
[BugFix] Fix Embedding Models with TP>1 (#5075)

robertgshaw2-redhat
authored and
Robert Shaw
committed
[Kernel][ROCm][AMD] Add fused_moe Triton configs for MI300X (#4951)

divakar-amd
authored and
Robert Shaw
committed
[Docs] Add Dropbox as sponsors (#5089)

simon-mo
authored and
Robert Shaw
committed
[Core] Consolidate prompt arguments to LLM engines (#4328)

authored and
Robert Shaw
committed
[Bugfix] Remove the last EOS token unless explicitly specified (#5077)

jsato8094
authored and
Robert Shaw
committed
[Misc] add gpu_memory_utilization arg (#5079)

pandyamarut
authored and
Robert Shaw
committed
[Core][Optimization] remove vllm-nccl (#5091)

youkaichao
authored and
Robert Shaw
committed
[Bugfix] Fix arguments passed to `Sequence` in stop checker test (#5092)

DarkLight1337
authored and
Robert Shaw
committed
[Core][Distributed] improve p2p access check (#4992)

youkaichao
authored and
Robert Shaw
committed
[Core] Cross-attention KV caching and memory-management (towards eventual encoder/decoder model support) (#4837)

afeldman-nm
authored and
Robert Shaw
committed
[Doc]Replace deprecated flag in readme (#4526)

ronensc
authored and
Robert Shaw
committed
[Bugfix][CI/Build] Fix test and improve code for `merge_async_iterators` (#5096)

DarkLight1337
authored and
Robert Shaw
committed
[Bugfix][CI/Build] Fix codespell failing to skip files in `git diff` (#5097)

DarkLight1337
authored and
Robert Shaw
committed
[Core] Avoid the need to pass `None` values to `Sequence.inputs` (#5099)

DarkLight1337
authored and
Robert Shaw
committed
[Bugfix] logprobs is not compatible with the OpenAI spec #4795 (#5031)

Etelis
authored and
Robert Shaw
committed
[Bugfix / Core] Prefix Caching Guards (merged with main) (#4846)

committed
[Bugfix] gptq_marlin: Ensure g_idx_sort_indices is not a Parameter (#5108)

alexm-redhat
authored and
Robert Shaw
committed
[CI/Build] Docker cleanup functionality for amd servers (#5112)

authored and
Robert Shaw
committed
[BUGFIX] [FRONTEND] Correct chat logprobs (#5029)

br3no
authored and
Robert Shaw
committed
[Bugfix] Automatically Detect SparseML models (#5119)

robertgshaw2-redhat
authored and
Robert Shaw
committed
[CI/Build] increase wheel size limit to 200 MB (#5130)

youkaichao
authored and
Robert Shaw
committed
[Misc] remove duplicate definition of `seq_lens_tensor` in model_runner.py (#5129)

ita9naiwa
authored and
Robert Shaw
committed
[Doc] Use intersphinx and update entrypoints docs (#5125)

DarkLight1337
authored and
Robert Shaw
committed
add doc about serving option on dstack (#3074)

authored and
Robert Shaw
committed
Bump version to v0.4.3 (#5046)

simon-mo
authored and
Robert Shaw
committed
[Build] Disable sm_90a in cu11 (#5141)

simon-mo
authored and
Robert Shaw
committed
[Bugfix] Avoid Warnings in SparseML Activation Quantization (#5120)

robertgshaw2-redhat
authored and
Robert Shaw
committed
[Kernel] Marlin_24: Ensure the mma.sp instruction is using the ::ordered_metadata modifier (introduced with PTX 8.5) (#5136)

alexm-redhat
authored and
Robert Shaw
committed
Fix cutlass sm_90a vesrion in CMakeList

simon-mo
authored and
Robert Shaw
committed
[Model] Support MAP-NEO model (#5081)

authored and
Robert Shaw
committed
Revert "[Kernel] Marlin_24: Ensure the mma.sp instruction is using the ::ordered_metadata modifier (introduced with PTX 8.5)" (#5149)

simon-mo
authored and
Robert Shaw
committed
[Misc]: optimize eager mode host time (#4196)

authored and
Robert Shaw
committed
[Model] Enable FP8 QKV in MoE and refine kernel tuning script (#5039)

comaniac
authored and
Robert Shaw
committed
[Doc] Add checkmark for GPTBigCodeForCausalLM LoRA support (#5171)

njhill
authored and
Robert Shaw
committed
[Build] Guard against older CUDA versions when building CUTLASS 3.x kernels (#5168)

tlrmchlsmth
authored and
Robert Shaw
committed
format
Robert Shaw
committed

Commits on Jun 9, 2024

skip blockspase attention
Robert Shaw
committed
fix falcon
Robert Shaw
committed
skip sliding window chunked prefill
Robert Shaw
committed
skip prefix prefill
Robert Shaw
committed
skip tensorizer
Robert Shaw
committed
[Misc][Breaking] Change FP8 checkpoint format from act_scale -> input_scale (#5353)

mgoin
authored and
Robert Shaw
committed
format
Robert Shaw
committed
fix issue with internal method
Robert Shaw
committed
formatting
Robert Shaw
committed
disabled more kernel tests that use triton
Robert Shaw
committed
updated cutlass skipping. We need cuda 12.4 in automation
Robert Shaw
committed
trigger kernel tests in automation
Robert Shaw
committed
cleanup spurious setup.py change
Robert Shaw
committed
readded the missing images
Robert Shaw
committed
multilora inference
Robert Shaw
committed
offline inference with prefix
Robert Shaw
committed
backend request func
Robert Shaw
committed
benchmark serving
Robert Shaw
committed
prod monitoring readme
Robert Shaw
committed
format
Robert Shaw
committed
fix benchmark issue - internal method changed
Robert Shaw
committed
removed skip for remote push edits
Robert Shaw
committed

Commits on Jun 10, 2024

update internal method in benchmark throughput too
Robert Shaw
committed
skip triton sampler tests
Robert Shaw
committed