[FS-Offloading] Batch Lookup in C by varun-sundar-rabindranath · Pull Request #46713 · vllm-project/vllm

varun-sundar-rabindranath · 2026-06-25T09:00:53Z

Purpose

TieredOffloading with FS tier on main yields poor performance. This is mostly due to lookup delays triggered by the FSAsyncLookupManager.
We find that the thread backed FSAsyncLookup is severely impacted by the GIL.

Benchmark

#!/bin/bash                                                                                                                                                                                                                                                                                                               
MODEL="google/gemma-4-31B-it"
TP_SIZE=2
CPU_BYTES=429496729600
NVME_FS_DIR="/mnt/nvme-storage/"

KV_TRANSFER_CONFIG=$(cat <<EOF
{
  "kv_connector": "OffloadingConnector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "spec_name": "TieringOffloadingSpec",
    "cpu_bytes_to_use": ${CPU_BYTES},
    "eviction_policy": "lru",
    "secondary_tiers": [{
      "type": "fs",
      "root_dir": "${NVME_FS_DIR}",
      "n_read_threads": 16,
      "n_write_threads": 16
    }]
  }
}
EOF
)

vllm serve "${MODEL}" \
        --chat-template  "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>\n'}}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant\n'}}{% endif %}" \
      --tensor-parallel-size="${TP_SIZE}" \
      --kv-transfer-config "${KV_TRANSFER_CONFIG}" \
      --enable-prefix-caching \
      --no-disable-hybrid-kv-cache-manager

TARGET_URL="http://127.0.0.1:8000"
BENCH_RATE="100"
BENCH_RATE_TYPE="concurrent"
BENCH_MAX_SECONDS="700"
BENCH_RANDOM_SEED="889"
BENCH_TURNS=5
BENCH_PROMPT_TOKENS="128"
BENCH_OUTPUT_TOKENS="128"
BENCH_PREFIX_TOKENS="10000"
PREFIX_COUNT=$((2 * BENCH_RATE))

DATA="{\"prompt_tokens\":${BENCH_PROMPT_TOKENS},\"output_tokens\":${BENCH_OUTPUT_TOKENS},\"prefix_tokens\":${BENCH_PREFIX_TOKENS},\"turns\":${BENCH_TURNS},\"prefix_count\":${PREFIX_COUNT}}"

guidellm benchmark run \
  --target="${TARGET_URL}" \
  --rate-type="${BENCH_RATE_TYPE}" \
  --rate="${BENCH_RATE}" \
  --max-seconds="${BENCH_MAX_SECONDS}" \
  --random-seed="${BENCH_RANDOM_SEED}" \
  --data="${DATA}" \
  --sample-requests=0

This PR:

ℹ Request Latency Statistics (Completed Requests)
|============|=========|========|=========|=========|=======|=======|=======|=======|
| Benchmark  | Request Latency || TTFT             || ITL          || TPOT         ||
| Strategy   | Sec             || ms               || ms           || ms           ||
|            | Mdn     | p95    | Mdn     | p95     | Mdn   | p95   | Mdn   | p95   |
|------------|---------|--------|---------|---------|-------|-------|-------|-------|
| concurrent | 31.9    | 85.7   | 15027.9 | 60035.7 | 158.1 | 231.6 | 249.2 | 669.2 |
|============|=========|========|=========|=========|=======|=======|=======|=======|


ℹ Server Throughput Statistics (All Requests)
|============|=======|=======|=========|==============|===============|==============|
| Benchmark  | Requests              ||| Input Tokens | Output Tokens | Total Tokens |
| Strategy   | Concurrency  || Per Sec | Per Sec      | Per Sec       | Per Sec      |
|            | Mdn   | Mean  | Mean                                               ||||
|------------|-------|-------|---------|--------------|---------------|--------------|
| concurrent | 100.0 | 100.0 | 1.8     | 19442.5      | 229.4         | 19671.9      |
|============|=======|=======|=========|==============|===============|==============|

Test Plan

Added unit tests

Test Result

Unit tests pass

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

orozery

Thanks @varun-sundar-rabindranath !
Looks mostly good to me.

Can we add some tests? Claude suggestion:

Unit test importing fsio_C.batch_lookup directly
Edge cases: empty list, single element, mix of existing/non-existing paths
Error case: list containing non-string (should raise TypeError)
Test the fallback path when C extension is unavailable

orozery · 2026-06-25T11:04:59Z

@@ -21,10 +21,17 @@
 from collections.abc import Iterable
 from typing import TYPE_CHECKING

+try:
+    from vllm.fsio_C import batch_lookup as batch_lookup_C


can we rename fsio -> fs_io (throughout)?

Made the change. The extensions are now fs_io_C 👍 this is a better name.

varun-sundar-rabindranath · 2026-06-25T16:58:52Z

Hi @orozery Thanks for the review. I have addressed all the comments. Can you take another pass please. Thanks 🙌

varun-sundar-rabindranath · 2026-06-25T16:58:59Z

@claude review

orozery · 2026-06-25T18:05:07Z

+
+  std::vector<int> exists_flags(n);
+  {
+    Py_BEGIN_ALLOW_THREADS _batch_lookup(paths, exists_flags);


nit:

Suggested change

Py_BEGIN_ALLOW_THREADS _batch_lookup(paths, exists_flags);

Py_BEGIN_ALLOW_THREADS

_batch_lookup(paths, exists_flags);

I have been struggling with clang-format for this. Looks like spinloop.cpp is also suffering from the same

vllm/csrc/spinloop.cpp

Line 149 in 96eb8dd

Py_BEGIN_ALLOW_THREADS _mm_mwaitx((1 << 1), 0,

orozery · 2026-06-25T18:06:14Z

+#
+# fsio extension (pure CXX; must stay above the non-CUDA device branch
+# so CPU builds define the target before the early return).
+# Provides batch faccessat(2) with a single GIL release for the entire batch,


Suggested change

# Provides batch faccessat(2) with a single GIL release for the entire batch,

# Provides batch access(2) with a single GIL release for the entire batch,

Removed references to batch_lookup and fixed the comment to be more generic as this module could be extended in the future.

varun-sundar-rabindranath · 2026-06-27T13:20:01Z

The two failing tests,
amd-lora-4-mi325-1
hybrid-ssm-nixlconnector-pd-prefix-cache-test-2-g
are unrelated and happen on the nightly as well.

nightly failure :
amd-lora-4-mi325-1 - https://buildkite.com/vllm/ci/builds/74801/list?jid=019f0860-e313-4eee-9b48-48c25873441b&tab=output
Hybrid SSM NixlConnector PD prefix cache test (2 GPUs) - https://buildkite.com/vllm/ci/builds/74801/list?sid=019f07ab-3364-4fdb-86d1-7da3b21eaeb9&tab=output

Signed-off-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com> Signed-off-by: <>

varun-sundar-rabindranath · 2026-06-28T13:05:50Z

Rebased to re-run the tests with latest main.

varun-sundar-rabindranath · 2026-06-29T12:43:46Z

Requesting force merge on this PR,
The failing tests are unrelated,

[buildkite/ci/pr/hybrid-ssm-nixlconnector-pd-prefix-cache-test-2-g](https://buildkite.com/vllm/ci/builds/74942#019f0e56-55cf-4fb1-a4ac-c79aaa666ebe) happens on nightly (same error) https://buildkite.com/vllm/ci/builds/75033/list?sid=019f11f7-e23d-4a61-9ef1-de575a4d41c1&tab=output
[buildkite/ci/pr/async-engine-inputs-utils-worker-config-cpu](https://buildkite.com/vllm/ci/builds/74942#019f0f1d-250a-464a-9419-dea234bddca2) buildkite/ci/pr/async-engine-inputs-utils-worker-config-cpu happens on main (same error) - https://buildkite.com/vllm/ci/builds/75089/canvas?jid=019f12e4-e5e5-4e8b-a39d-7177d5084042&tab=output

cc @orozery @tlrmchlsmth

Signed-off-by: <> Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com> Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

Signed-off-by: <> Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>

### What this PR does / why we need it? refer to: vllm-project/vllm#40020 , vllm-project/vllm#49152 , vllm-project/vllm#46713 and vllm-project/vllm#49734 , add multi-tier KV cache offloading framework, reusing the c operator from the upstream vllm for batch loading, saving, and searching. ### How was this patch tested? test reuslt: |num-clients/max-active-conversations|base(without offload, TTFT/TPOT)|with dram(TTFT/TPOT)|with dram+ssd(TTFT/TPOT)| | :---:|:---:|:---:|:---:| |8/24|1151.21/87.28|863.02/67.62|605.81/49.87| |16/48|7251.86/108.90|5679.01/83.75|3667.07/59.05| model script: ``` export TP=1 export MODEL_PATH=/nas/disk1/Qwen3-14B export MODEL_NAME=Qwen3-14B export PORT=10113 #export CUDA_VISIBLE_DEVICES=3 export ASCEND_RT_VISIBLE_DEVICES=4 python3 -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port ${PORT} --dtype bfloat16 --model ${MODEL_PATH} --served-model-name ${MODEL_NAME} --tensor-parallel-size ${TP} --gpu-memory-utilization 0.6 --no-enable-prefix-caching --max-model-len 32768 --trust-remote-code --kv-transfer-config '{ "kv_connector": "OffloadingConnector", "kv_role": "kv_both", "kv_connector_extra_config": { "spec_name": "TieringOffloadingSpec", "cpu_bytes_to_use": 10737418240, "block_size": 128, "eviction_policy": "lru", "secondary_tiers": [ { "type": "fs", "root_dir": "/mnt/kv_cache4", "n_read_threads": 32, "n_write_threads": 16 } ] } }' ``` test script: ``` export MODEL_NAME=/nas/disk1/Qwen3-14B python /model/vllm/benchmarks/multi_turn/benchmark_serving_multi_turn.py --url http://127.0.0.1:10113 --model $MODEL_NAME --served-model-name Qwen3-14B --seed 1234 --input-file /model/vllm/benchmarks/multi_turn/generate_multi_turn2.json \ --num-clients 8 --max-active-conversations 24 ``` generate_multi_turn2.json ``` { "filetype": "generate_conversations", "num_conversations": 96, "text_files": ["pg1184.txt"], "print_stats": false, "prompt_input": { "num_turns": { "distribution": "uniform", "min": 12, "max": 18 }, "common_prefix_num_tokens": { "distribution": "constant", "value": 2000 }, "prefix_num_tokens": { "distribution": "lognormal", "average": 2000, "max": 10000 }, "num_tokens": { "distribution": "uniform", "min": 240, "max": 320 } }, "prompt_output": { "num_tokens": { "distribution": "uniform", "min": 80, "max": 120 } } } ``` - vLLM main: vllm-project/vllm@e6bfe03 --------- Signed-off-by: HF-001 <1670186653@qq.com>

…10575) ### What this PR does / why we need it? refer to: vllm-project/vllm#40020 , vllm-project/vllm#49152 , vllm-project/vllm#46713 and vllm-project/vllm#49734 , add multi-tier KV cache offloading framework, reusing the c operator from the upstream vllm for batch loading, saving, and searching. ### How was this patch tested? test reuslt: |num-clients/max-active-conversations|base(without offload, TTFT/TPOT)|with dram(TTFT/TPOT)|with dram+ssd(TTFT/TPOT)| | :---:|:---:|:---:|:---:| |8/24|1151.21/87.28|863.02/67.62|605.81/49.87| |16/48|7251.86/108.90|5679.01/83.75|3667.07/59.05| model script: ``` export TP=1 export MODEL_PATH=/nas/disk1/Qwen3-14B export MODEL_NAME=Qwen3-14B export PORT=10113 #export CUDA_VISIBLE_DEVICES=3 export ASCEND_RT_VISIBLE_DEVICES=4 python3 -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port ${PORT} --dtype bfloat16 --model ${MODEL_PATH} --served-model-name ${MODEL_NAME} --tensor-parallel-size ${TP} --gpu-memory-utilization 0.6 --no-enable-prefix-caching --max-model-len 32768 --trust-remote-code --kv-transfer-config '{ "kv_connector": "OffloadingConnector", "kv_role": "kv_both", "kv_connector_extra_config": { "spec_name": "TieringOffloadingSpec", "cpu_bytes_to_use": 10737418240, "block_size": 128, "eviction_policy": "lru", "secondary_tiers": [ { "type": "fs", "root_dir": "/mnt/kv_cache4", "n_read_threads": 32, "n_write_threads": 16 } ] } }' ``` test script: ``` export MODEL_NAME=/nas/disk1/Qwen3-14B python /model/vllm/benchmarks/multi_turn/benchmark_serving_multi_turn.py --url http://127.0.0.1:10113 --model $MODEL_NAME --served-model-name Qwen3-14B --seed 1234 --input-file /model/vllm/benchmarks/multi_turn/generate_multi_turn2.json \ --num-clients 8 --max-active-conversations 24 ``` generate_multi_turn2.json ``` { "filetype": "generate_conversations", "num_conversations": 96, "text_files": ["pg1184.txt"], "print_stats": false, "prompt_input": { "num_turns": { "distribution": "uniform", "min": 12, "max": 18 }, "common_prefix_num_tokens": { "distribution": "constant", "value": 2000 }, "prefix_num_tokens": { "distribution": "lognormal", "average": 2000, "max": 10000 }, "num_tokens": { "distribution": "uniform", "min": 240, "max": 320 } }, "prompt_output": { "num_tokens": { "distribution": "uniform", "min": 80, "max": 120 } } } ``` - vLLM main: vllm-project/vllm@e6bfe03 --------- Signed-off-by: HF-001 <1670186653@qq.com>

…10575) ### What this PR does / why we need it? refer to: vllm-project/vllm#40020 , vllm-project/vllm#49152 , vllm-project/vllm#46713 and vllm-project/vllm#49734 , add multi-tier KV cache offloading framework, reusing the c operator from the upstream vllm for batch loading, saving, and searching. ### How was this patch tested? test reuslt: |num-clients/max-active-conversations|base(without offload, TTFT/TPOT)|with dram(TTFT/TPOT)|with dram+ssd(TTFT/TPOT)| | :---:|:---:|:---:|:---:| |8/24|1151.21/87.28|863.02/67.62|605.81/49.87| |16/48|7251.86/108.90|5679.01/83.75|3667.07/59.05| model script: ``` export TP=1 export MODEL_PATH=/nas/disk1/Qwen3-14B export MODEL_NAME=Qwen3-14B export PORT=10113 #export CUDA_VISIBLE_DEVICES=3 export ASCEND_RT_VISIBLE_DEVICES=4 python3 -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port ${PORT} --dtype bfloat16 --model ${MODEL_PATH} --served-model-name ${MODEL_NAME} --tensor-parallel-size ${TP} --gpu-memory-utilization 0.6 --no-enable-prefix-caching --max-model-len 32768 --trust-remote-code --kv-transfer-config '{ "kv_connector": "OffloadingConnector", "kv_role": "kv_both", "kv_connector_extra_config": { "spec_name": "TieringOffloadingSpec", "cpu_bytes_to_use": 10737418240, "block_size": 128, "eviction_policy": "lru", "secondary_tiers": [ { "type": "fs", "root_dir": "/mnt/kv_cache4", "n_read_threads": 32, "n_write_threads": 16 } ] } }' ``` test script: ``` export MODEL_NAME=/nas/disk1/Qwen3-14B python /model/vllm/benchmarks/multi_turn/benchmark_serving_multi_turn.py --url http://127.0.0.1:10113 --model $MODEL_NAME --served-model-name Qwen3-14B --seed 1234 --input-file /model/vllm/benchmarks/multi_turn/generate_multi_turn2.json \ --num-clients 8 --max-active-conversations 24 ``` generate_multi_turn2.json ``` { "filetype": "generate_conversations", "num_conversations": 96, "text_files": ["pg1184.txt"], "print_stats": false, "prompt_input": { "num_turns": { "distribution": "uniform", "min": 12, "max": 18 }, "common_prefix_num_tokens": { "distribution": "constant", "value": 2000 }, "prefix_num_tokens": { "distribution": "lognormal", "average": 2000, "max": 10000 }, "num_tokens": { "distribution": "uniform", "min": 240, "max": 320 } }, "prompt_output": { "num_tokens": { "distribution": "uniform", "min": 80, "max": 120 } } } ``` - vLLM main: vllm-project/vllm@e6bfe03 --------- Signed-off-by: HF-001 <1670186653@qq.com> Signed-off-by: like-0517 <ithwlike@126.com>

varun-sundar-rabindranath requested review from ApostaC, Harry-Chen, LucasWilkinson, khluu, orozery and tlrmchlsmth as code owners June 25, 2026 09:00

claude Bot reviewed Jun 25, 2026

View reviewed changes

varun-sundar-rabindranath mentioned this pull request Jun 25, 2026

[FS Offloading] Move FSAsyncLookup into a subprocess #46559

Closed

mergify Bot added ci/build v1 labels Jun 25, 2026

varun-sundar-rabindranath mentioned this pull request Jun 25, 2026

[TieredOffloading] Bound Secondary Tier lookup times #45765

Closed

orozery reviewed Jun 25, 2026

View reviewed changes

orozery approved these changes Jun 25, 2026

View reviewed changes

orozery added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 25, 2026

varun-sundar-rabindranath force-pushed the varun/c-async-lookup branch 3 times, most recently from 687eb7f to 0766309 Compare June 27, 2026 03:19

Varun Sundar Rabindranath added 3 commits June 28, 2026 09:03

Move batch lookups to C

cc2e953

Signed-off-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com> Signed-off-by: <>

fix comment

9ecea40

Signed-off-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com> Signed-off-by: <>

fix test

6d18de5

Signed-off-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com> Signed-off-by: <>

varun-sundar-rabindranath force-pushed the varun/c-async-lookup branch from 0766309 to 6d18de5 Compare June 28, 2026 13:04

hwx707 mentioned this pull request Jun 29, 2026

[Core] Add scheduler-orchestrated worker transfers for multi-node KV cache tiering #46556

Open

mgoin approved these changes Jun 29, 2026

View reviewed changes

vllm-bot merged commit c8fb296 into vllm-project:main Jun 29, 2026
226 of 229 checks passed

jethac mentioned this pull request Jul 1, 2026

[Bug]: Nemotron-3-Super-120B NVFP4 generation degenerates on recent main — bisected to #46756 #47241

Closed

2 tasks

HF-001 mentioned this pull request Aug 31, 2026

[Feature] Add multi-tier KV cache offloading framework vllm-project/vllm-ascend#10575

Merged

	Py_BEGIN_ALLOW_THREADS _batch_lookup(paths, exists_flags);
	Py_BEGIN_ALLOW_THREADS
	_batch_lookup(paths, exists_flags);

	# Provides batch faccessat(2) with a single GIL release for the entire batch,
	# Provides batch access(2) with a single GIL release for the entire batch,

Uh oh!

Conversation

varun-sundar-rabindranath commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Benchmark

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

orozery left a comment

Choose a reason for hiding this comment

Uh oh!

orozery Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

varun-sundar-rabindranath Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

varun-sundar-rabindranath commented Jun 25, 2026

Uh oh!

varun-sundar-rabindranath commented Jun 25, 2026

Uh oh!

orozery Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

varun-sundar-rabindranath Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

orozery Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

varun-sundar-rabindranath Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

varun-sundar-rabindranath commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

varun-sundar-rabindranath commented Jun 28, 2026

Uh oh!

varun-sundar-rabindranath commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

varun-sundar-rabindranath commented Jun 25, 2026 •

edited

Loading

varun-sundar-rabindranath commented Jun 27, 2026 •

edited

Loading