This repository was archived by the owner on Oct 11, 2024. It is now read-only.

[Rel Eng] Upstream sync 2024 06 11#298

Merged

robertgshaw2-redhat merged 93 commits intomainfrom

upstream-sync-2024-06-11

Jun 11, 2024

Commits on Jun 11, 2024

[CI/Build] CMakeLists: build all extensions' cmake targets at the same time (#5034)

dtrifiro
authored and
Robert Shaw
committed
[Kernel] Refactor CUTLASS kernels to always take scales that reside on the GPU (#5137)

tlrmchlsmth
authored and
Robert Shaw
committed
[Kernel] Update Cutlass fp8 configs (#5144)

authored and
Robert Shaw
committed
[Minor] Fix the path typo in loader.py: save_sharded_states.py -> save_sharded_state.py (#5151)

dashanji
authored and
Robert Shaw
committed
[Bugfix] Fix call to init_logger in openai server (#4765)

NadavShmayo
authored and
Robert Shaw
committed
[Feature][Kernel] Support bitsandbytes quantization and QLoRA (#4776)

chenqianfzh
authored and
Robert Shaw
committed
[Bugfix] Remove deprecated @abstractproperty (#5174)

zhuohan123
authored and
Robert Shaw
committed
[Bugfix]: Fix issues related to prefix caching example (#5177) (#5180)

Delviet
authored and
Robert Shaw
committed
[BugFix] Prevent `LLM.encode` for non-generation Models (#5184)

authored and
Robert Shaw
committed
Update test_ignore_eos (#4898)

simon-mo
authored and
Robert Shaw
committed
[Frontend][OpenAI] Support for returning max_model_len on /v1/models response (#4643)

Avinash-Raj
authored and
Robert Shaw
committed
[Kernel][ROCm][AMD] enable fused topk_softmax kernel for moe layer (#4927)

divakar-amd
authored and
Robert Shaw
committed
[Misc] Simplify code and fix type annotations in `conftest.py` (#5118)

DarkLight1337
authored and
Robert Shaw
committed
[Core] Support image processor (#4197)

DarkLight1337
authored and
Robert Shaw
committed
[Core] Remove unnecessary copies in flash attn backend (#5138)

Yard1
authored and
Robert Shaw
committed
[Kernel] Pass a device pointer into the quantize kernel for the scales (#5159)

tlrmchlsmth
authored and
Robert Shaw
committed
[CI/BUILD] enable intel queue for longer CPU tests (#4113)

zhouyuan
authored and
Robert Shaw
committed
[Misc]: Implement CPU/GPU swapping in BlockManagerV2 (#3834)

Kaiyang-Chen
authored and
Robert Shaw
committed
New CI template on AWS stack (#5110)

khluu
authored and
Robert Shaw
committed
[FRONTEND] OpenAI `tools` support named functions (#5032)

br3no
authored and
Robert Shaw
committed
[Bugfix] Support `prompt_logprobs==0` (#5217)

toslunar
authored and
Robert Shaw
committed
[Bugfix] Add warmup for prefix caching example (#5235)

zhuohan123
authored and
Robert Shaw
committed
[Kernel] Enhance MoE benchmarking & tuning script (#4921)

WoosukKwon
authored and
Robert Shaw
committed
[Bugfix]: During testing, use pytest monkeypatch for safely overriding the env var that indicates the vLLM backend (#5210)

afeldman-nm
authored and
Robert Shaw
committed
[Bugfix] Fix torch.compile() error when using MultiprocessingGPUExecutor (#5229)

zifeitong
authored and
Robert Shaw
committed
[CI/Build] Add inputs tests (#5215)

DarkLight1337
authored and
Robert Shaw
committed
[Bugfix] Fix a bug caused by pip install setuptools>=49.4.0 for CPU backend (#5249)

DamonFool
authored and
Robert Shaw
committed
[Kernel] Add back batch size 1536 and 3072 to MoE tuning (#5242)

WoosukKwon
authored and
Robert Shaw
committed
[CI/Build] Simplify model loading for `HfRunner` (#5251)

DarkLight1337
authored and
Robert Shaw
committed
[CI/Build] Reducing CPU CI execution time (#5241)

bigPYJ1151
authored and
Robert Shaw
committed
[CI] mark AMD test as softfail to prevent blockage (#5256)

simon-mo
authored and
Robert Shaw
committed
[Misc] Add transformers version to collect_env.py (#5259)

mgoin
authored and
Robert Shaw
committed
[Misc] update collect env (#5261)

youkaichao
authored and
Robert Shaw
committed
[Bugfix] Fix prompt_logprobs when SamplingParams.detokenize is set to True (#5226)

zifeitong
authored and
Robert Shaw
committed
[Misc] Add CustomOp interface for device portability (#5255)

WoosukKwon
authored and
Robert Shaw
committed
[Misc] Fix docstring of get_attn_backend (#5271)

WoosukKwon
authored and
Robert Shaw
committed
[Frontend] OpenAI API server: Add `add_special_tokens` to ChatCompletionRequest (default False) (#5278)

tomeras91
authored and
Robert Shaw
committed
[CI] Add nightly benchmarks (#5260)

simon-mo
authored and
Robert Shaw
committed
[misc] benchmark_serving.py -- add ITL results and tweak TPOT results (#5263)

tlrmchlsmth
authored and
Robert Shaw
committed
[Kernel] Add GPU architecture guards to the CUTLASS w8a8 kernels to reduce binary size (#5157)

authored and
Robert Shaw
committed
[Model] Correct Mixtral FP8 checkpoint loading (#5231)

comaniac
authored and
Robert Shaw
committed
[BugFix] Apply get_cached_tokenizer to the tokenizer setter of LLM (#5207)

authored and
Robert Shaw
committed
[Kernel] Re-tune Mixtral MoE configurations for FP8 on H100 (#5238)

pcmoritz
authored and
Robert Shaw
committed
[Docs] Add Sequoia as sponsors (#5287)

simon-mo
authored and
Robert Shaw
committed
[Speculative Decoding] Add `ProposerWorkerBase` abstract class (#5252)

njhill
authored and
Robert Shaw
committed
[BugFix] Fix log message about default max model length (#5284)

njhill
authored and
Robert Shaw
committed
[Bugfix] Make EngineArgs use named arguments for config construction (#5285)

mgoin
authored and
Robert Shaw
committed
[Bugfix][Frontend/Core] Don't log exception when AsyncLLMEngine gracefully shuts down. (#5290)

wuisawesome
authored and
Robert Shaw
committed
[Misc] Skip for logits_scale == 1.0 (#5291)

WoosukKwon
authored and
Robert Shaw
committed
[Docs] Add Ray Summit CFP (#5295)

simon-mo
authored and
Robert Shaw
committed
[CI] Disable flash_attn backend for spec decode (#5286)

simon-mo
authored and
Robert Shaw
committed
[Frontend][Core] Update Outlines Integration from `FSM` to `Guide` (#4109)

authored and
Robert Shaw
committed
[CI/Build] Update vision tests (#5307)

DarkLight1337
authored and
Robert Shaw
committed
Bugfix: fix broken of download models from modelscope (#5233)

authored and
Robert Shaw
committed
[Kernel] Retune Mixtral 8x22b configs for FP8 on H100 (#5294)

pcmoritz
authored and
Robert Shaw
committed
[Frontend] enable passing multiple LoRA adapters at once to generate() (#5300)

mgoldey
authored and
Robert Shaw
committed
[Core] Avoid copying prompt/output tokens if no penalties are used (#5289)

Yard1
authored and
Robert Shaw
committed
[Core] Change LoRA embedding sharding to support loading methods (#5038)

Yard1
authored and
Robert Shaw
committed
[Misc] Missing error message for custom ops import (#5282)

DamonFool
authored and
Robert Shaw
committed
[Feature][Frontend]: Add support for `stream_options` in `ChatCompletionRequest` (#5135)

Etelis
authored and
Robert Shaw
committed
[Misc][Utils] allow get_open_port to be called for multiple times (#5333)

youkaichao
authored and
Robert Shaw
committed
[Kernel] Switch fp8 layers to use the CUTLASS kernels (#5183)

tlrmchlsmth
authored and
Robert Shaw
committed
Remove Ray health check (#4693)

Yard1
authored and
Robert Shaw
committed
Addition of lacked ignored_seq_groups in _schedule_chunked_prefill (#5296)

JamesLim-sy
authored and
Robert Shaw
committed
[Kernel] Dynamic Per-Token Activation Quantization (#5037)

authored and
Robert Shaw
committed
[Frontend] Add OpenAI Vision API Support (#5237)

authored and
Robert Shaw
committed
[Misc] Remove unused cuda_utils.h in CPU backend (#5345)

DamonFool
authored and
Robert Shaw
committed
fix DbrxFusedNormAttention missing cache_config (#5340)

authored and
Robert Shaw
committed
[Bug Fix] Fix the support check for FP8 CUTLASS (#5352)

cli99
authored and
Robert Shaw
committed
[Misc] Add args for selecting distributed executor to benchmarks (#5335)

BKitor
authored and
Robert Shaw
committed
[ROCm][AMD] Use pytorch sdpa math backend to do naive attention (#4965)

hongxiayang
authored and
Robert Shaw
committed
[CI/Test] improve robustness of test (hf_runner) (#5347)

youkaichao
authored and
Robert Shaw
committed
[CI/Test] improve robustness of test (vllm_runner) (#5357)

youkaichao
authored and
Robert Shaw
committed
[Misc][Breaking] Change FP8 checkpoint format from act_scale -> input_scale (#5353)

mgoin
authored and
Robert Shaw
committed
[Core][CUDA Graph] add output buffer for cudagraph (#5074)

youkaichao
authored and
Robert Shaw
committed
[mis][ci/test] fix flaky test in test_sharded_state_loader.py (#5361)

youkaichao
authored and
Robert Shaw
committed
[Kernel][Misc] Use TORCH_LIBRARY instead of PYBIND11_MODULE for custom ops (#5047)

bnellnm
authored and
Robert Shaw
committed
[Bugfix] Fix KeyError: 1 When Using LoRA adapters (#5164)

tdrunsri
authored and
Robert Shaw
committed
[Misc] Update to comply with the new `compressed-tensors` config (#5350)

authored and
Robert Shaw
committed
[Frontend][Misc] Enforce Pixel Values as Input Type for VLMs in API Server (#5374)

ywang96
authored and
Robert Shaw
committed
[misc][typo] fix typo (#5372)

youkaichao
authored and
Robert Shaw
committed
[Misc] Improve error message when LoRA parsing fails (#5194)

DarkLight1337
authored and
Robert Shaw
committed
[Model] Initial support for LLaVA-NeXT (#4199)

authored and
Robert Shaw
committed
[Feature][Frontend]: Continued `stream_options` implementation also in CompletionRequest (#5319)

Etelis
authored and
Robert Shaw
committed
[Bugfix] Fix LLaVA-NeXT (#5380)

DarkLight1337
authored and
Robert Shaw
committed
[ci] Use small_cpu_queue for doc build (#5331)

khluu
authored and
Robert Shaw
committed
[ci] Mount buildkite agent on Docker container to upload benchmark results (#5330)

khluu
authored and
Robert Shaw
committed
[Docs] Add Docs on Limitations of VLM Support (#5383)

ywang96
authored and
Robert Shaw
committed
[Docs] Alphabetically sort sponsors (#5386)

WoosukKwon
authored and
Robert Shaw
committed
Bump version to v0.5.0 (#5384)

simon-mo
authored and
Robert Shaw
committed
format
Robert Shaw
committed
updated test model logprobs
Robert Shaw
committed
format
Robert Shaw
committed