[khluu]

Overview

Re-enable the KV connector CI jobs disabled in #63609 (follow-up to #63608 and #64010). Instead of installing NIXL, LMCache, Mooncake and CuPy at job time, the CUDA 13 CI image now bakes them from a hash-pinned lock.

Claims

• The CI test image installs the KV connector packages in the test-deps stage with uv pip install --require-hashes --no-deps from requirements/test/kv_connectors_cuda.txt. That lock is compiled against requirements/test/cuda.txt and regenerated by a new pip-compile-kv-connectors pre-commit hook. At build time it adds 16 packages and changes none already in the image.
• install-kv-connectors.sh becomes check-kv-connectors.sh. It only verifies the packages are present, so it can no longer resolve packages at job time.
• ROCm needs no image change. kv_connectors_rocm.txt (tblib, lm_eval[api]) is fully covered by requirements/test/rocm.txt, so the runtime installs were no-ops. They and the file are removed.
• 26 job blocks re-enabled. The CUDA and AMD jobs are listed in the build below.
• One missed runtime install removed: "Mooncake EC TCP E2E" ran uv pip install mooncake-transfer-engine>=0.3.12 into a venv. It now uses the image's Mooncake through --system-site-packages.
• 6 tethering-allowlist entries removed.

Validation

• pre-commit run on all changed files passed: pip-compile-kv-connectors, shellcheck, step keys, test tethering (--all reports nothing stale) and Dockerfile graph.
• All Buildkite YAML parses. Each restored block matches its pre-[CI] Disable jobs with unpinned runtime installs #63609 definition except for the script rename and the dropped ROCm install line.
• Cooldown: every newly introduced package was published at least 7 days ago except nixl/nixl-cu13 1.5.2 (PyPI Oct 29). Main already pins it in requirements/kv_connectors.txt since [NIXL][CI/Build] Bump NIXL to 1.5.2 and drop source build for Rubin #64585. Please don't merge before Nov 5 if we hold to the 7-day rule.
• CI: a selected-steps build of the image plus the re-enabled jobs is linked in a comment below.

Details

• The image layer installs only the lock entries the image doesn't already have, with --require-hashes --no-deps. It then fails the build if any preinstalled package changed. This guard caught cuda-python/nvtx drift: the runtime base installs versions that requirements/test/cuda.txt doesn't pin. NCCL is excluded because the image pins it with its own NCCL_VERSION/UV_OVERRIDE.
• Scope: only the stable x86_64 CUDA 13 image. Torch-nightly and arm64 images don't get the packages, and check-kv-connectors.sh fails fast there with a clear message. The KV jobs only run on x86 L4/H100/H200/B200 pools.
• Still disabled:
  • A100 "NixlConnector PD + Spec Decode acceptance": the A100 queue was retired in [CI] Remove the retired A100 queue's steps; keep their live AMD mirrors standalone #64602. Its MI355 mirror needs a standalone conversion.
  • The two Intel jobs that pip install -r requirements/kv_connectors.txt on the XPU image. They need an Intel image change.
• The release images (INSTALL_KV_CONNECTORS=true) are unchanged.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[khluu]

Claude Code here (AI assistant working with @khluu). Selected-steps CI run for this PR: the image build plus the 20 re-enabled CUDA KV connector jobs and the Mooncake EC job: https://buildkite.com/vllm/ci/builds/99153

[khluu]

Claude Code here (AI assistant working with @khluu). Build 99153 failed at image build: the image pins NCCL 2.31.2 through a global UV_OVERRIDE, which collided with the lock's nvidia-nccl-cu13==2.30.7 under --require-hashes. Fixed in ca77a3f: NCCL is no longer emitted in the lock (the image owns it), and the install step now fails if it changes any package that was already in the image. Re-run: https://buildkite.com/vllm/ci/builds/99164

[khluu]

Claude Code here (AI assistant working with @khluu). Status of selected-steps build https://buildkite.com/vllm/ci/builds/99176:

• Image build passes. The KV connector layer installs 16 hash-verified packages and changes nothing already in the image.
• V1 KV Connectors shard 2/3 fails on test_nixl_connector.py::test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups]. kv_caches_base_addr holds the packed raw base twice ([raw, raw, ring] instead of [raw, ring]); the other 894 tests pass. This is a latent failure on main, not caused by this PR: the job has been disabled since [CI] Disable jobs with unpinned runtime installs #63609 (Oct 25), and both [Bugfix][NIXL] Dedup transfer regions by (base_addr, block_len) #57662 (dedup regions by (base_addr, block_len), Oct 29) and [KV Connector][NIXL] Pair KV regions by layer for every transfer #63416 (pair KV regions by layer, which edited this test, Oct 31) merged while it couldn't run. Which side is right, the dedup behaviour or the assertion, is a NIXL owners' call, so I haven't changed it here.

[khluu]

Claude Code here (AI assistant working with @khluu). Full triage of https://buildkite.com/vllm/ci/builds/99176. None of the failures is install-related; all of them landed on main while these jobs were disabled (Oct 25 to now):

• Kimi K3 Prefix Cache (3/4 shards): the 4 dcp2-*-dspark cases now report XPASS(strict), because the FlashInfer MLA DCP fix ([Bugfix][Attention] Fix FlashInfer MLA DCP decode on ragged spec-decode batches #59392) merged on Oct 27. I dropped the stale xfail in cc47b9e; re-run: https://buildkite.com/vllm/ci/builds/99200
• V1 KV Connectors shard 2/3: the NIXL test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups] failure described above.
• V1 KV Connectors shard 3/3 (6 of 1107 failed):
  • offloading_connector/test_scheduler.py: test_partial_tail_store_uses_attention_and_recurrent_cow_sources and both test_eagle_partial_tail_round_trip_serves_mamba_checkpoint variants. These follow [KV Offload] Emit self-describing events for recurrent state groups #62613, [Bugfix][KV Offload] Preserve stored event prefix order #63249, [Bugfix][KV Offload] Deduplicate in-flight partial-tail loads #64400 and [Perf][KV Offload] Keep offload event metadata out of the cyclic GC #63927.
  • test_mooncake_connector.py: test_register_worker_retries_on_read_timeout and test_register_worker_raises_after_max_attempts. The mocked timeouts aren't reached, so the code hits real ConnectErrors. These follow [KV Connector][Mooncake] Renew prefill KV leases from the decoder #64193, [Perf][KV Connector][Mooncake] Vectorize transfer descriptor build #62478 and [KVConnector][Mooncake] Coalesce descending block ids to reduce descriptor number for Mooncake TE #63142.
  • test_mooncake_stats.py::test_expired_request_bumps_counter: MooncakeConnectorWorker lacks _kv_lease_duration. The test builds the worker without the new lease setup from [KV Connector][Mooncake] Renew prefill KV leases from the decoder #64193.

These need the respective owners to decide whether the code or the test is right. I haven't touched them here.

[orozery]

• offloading_connector/test_scheduler.py: test_partial_tail_store_uses_attention_and_recurrent_cow_sources and both test_eagle_partial_tail_round_trip_serves_mamba_checkpoint variants. These follow [KV Offload] Emit self-describing events for recurrent state groups #62613, [Bugfix][KV Offload] Preserve stored event prefix order #63249, [Bugfix][KV Offload] Deduplicate in-flight partial-tail loads #64400 and [Perf][KV Offload] Keep offload event metadata out of the cyclic GC #63927.

Fixed in #64993.

[khluu]

Claude Code here (on behalf of @khluu). Thanks @orozery. I merged main, which now includes #64993, into this branch (e962258) and am re-running V1 KV Connectors in https://buildkite.com/vllm/ci/builds/99534 to confirm the three offloading-connector scheduler tests pass. The Mooncake and NIXL failures listed above are separate and still unresolved.

🤖 Addressed by Claude Code

[khluu]

Claude Code here (on behalf of @khluu). V1 KV Connectors results after merging main, from https://buildkite.com/vllm/ci/builds/99534:

• Shard 3/3: the three offloading_connector/test_scheduler.py tests now pass, so [Test][KV Offload] Update stale partial-tail scheduler expectations #64993 fixed them. Thanks @orozery. The 3 failures left are all Mooncake and also fail on main:
  • test_register_worker_retries_on_read_timeout and test_register_worker_raises_after_max_attempts: the mocked httpx timeouts never fire, so the tests hit real errors. [CI/Build][KV Connector] Mooncake tests: patch the httpx module the connector uses #63122 looks like the fix.
  • test_mooncake_stats.py::test_expired_request_bumps_counter: MooncakeConnectorWorker has no _kv_lease_duration. The test builds the worker without the lease setup from [KV Connector][Mooncake] Renew prefill KV leases from the decoder #64193. I found no open fix for this one.
• Shard 2/3: the only failure (1 of 896) is the NIXL test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups], also failing on main. [Bugfix][NIXL] Preserve packed-row aliases during region dedup #64661 (preserve packed-row aliases during region dedup) addresses it.

None of these are caused by this PR, so I haven't patched them here.

🤖 Addressed by Claude Code