Support FP8-E5M2 KV Cache#2279

Merged

zhuohan123 merged 43 commits intovllm-project:mainfrom

zhaoyang-star:fp8_cache

Jan 29, 2024

Commits on Dec 27, 2023

test_cache.py passed
zhaoyang
committed
format
zhaoyang-star
committed
fix compiling warning
zhaoyang-star
committed

Commits on Dec 28, 2023

fix fp8x4 -> float4
zhaoyang-star
committed

Commits on Dec 29, 2023

test_attention pass
zhaoyang-star
committed
fix typo
zhaoyang-star
committed
fix typo
zhaoyang-star
committed
add latency & throughput benchmark
zhaoyang-star
committed
fix benchmark_latency.py
zhaoyang-star
committed

Commits on Jan 3, 2024

Merge branch 'main' into fp8_cache
zhaoyang-star
committed
fix copy_blocks
zhaoyang-star
committed
use VLLM_DISPATCH_CASE_FLOATING_BYTE_TYPES
zhaoyang-star
committed

Commits on Jan 12, 2024

add default behavior in description
zhaoyang-star
committed

Commits on Jan 13, 2024

grace code regarding to comments
zhaoyang-star
committed
add namespace fp8_e5m2_unscaled
zhaoyang-star
committed

Commits on Jan 15, 2024

change interface
zhaoyang-star
committed
solve conflict
zhaoyang-star
committed
fix tp
zhaoyang-star
committed
print log.info
zhaoyang-star
committed

Commits on Jan 16, 2024

add log and raise error on amd gpu
zhaoyang-star
committed

Commits on Jan 18, 2024

fix none error
zhaoyang-star
committed
do not use VLLM_LDG
zhaoyang-star
committed

Commits on Jan 19, 2024

fix mirror typo
zhaoyang-star
committed
fix compiler error on lower cc
zhaoyang-star
committed
fix unittest error
zhaoyang-star
committed
fix utest
zhaoyang-star
committed

Commits on Jan 20, 2024

Merge branch 'main' into fp8_cache
zhaoyang-star
committed

Commits on Jan 21, 2024

fix tp error
zhaoyang-star
committed

Commits on Jan 24, 2024

more clear, tp error
zhaoyang-star
committed
fix conflict
zhaoyang-star
committed
avoid compile fp8 when cuda version is lower than 11.8
zhaoyang-star
committed

Commits on Jan 25, 2024

fix ut
zhaoyang-star
committed
fix typo
zhaoyang-star
committed
loose NUM_BLOCKS in test_attention
zhaoyang-star
committed
update test_cache
zhaoyang-star
committed
fix yapf
zhaoyang-star
committed

Commits on Jan 26, 2024

loose num_blocks
zhaoyang-star
committed
fix stuck when tp>1
zhaoyang-star
committed

Commits on Jan 27, 2024

update
zhaoyang-star
committed
rename: create_kv_caches -> create_kv_caches_with_random
zhaoyang-star
committed

Commits on Jan 28, 2024

remove cache_dtype_str
zhaoyang-star
committed
add co-author

zhaoyang-star
and
zhuohan123
committed
Merge branch 'main' into fp8_cache
zhuohan123
committed