feat(vllm): support KV-transfer connectors with external store reset on refit

[aoshen02]

What does this PR do?

Enables vLLM KV-transfer connectors (e.g. MooncakeStoreConnector) for rollout, with the RL-correct hard reset of the external KV store on every weight refit. Two small changes, no new config schema:

1. Async engine dict coercion: vllm_kwargs.kv_transfer_config already reaches the sync engine (vllm.LLM converts dict → KVTransferConfig), but AsyncEngineArgs has no dict coercion — _create_engine now converts it explicitly, mirroring the existing compilation_config workaround.
2. External store reset on refit: all four prefix-cache reset sites (reset_prefix_cache/sleep, sync and async) now pass reset_connector=True. This cascades to connector.reset_cache() (for Mooncake: a global remove_all(force=True) on the master), so KV blocks computed with the previous policy weights are never served after a refit. Without a connector configured, reset_connector=True is an upstream-documented no-op (scheduler.py reset_connector_cache: "No connector attached -> nothing to reset, treat as success").

All reset sites run at step boundaries after rollout drain, satisfying the Mooncake store scheduler's no-in-flight-transfer contract. Multiple replicas resetting the same master is idempotent. The pinned vllm==0.25.1 contains everything needed (MooncakeStoreConnector since v0.22, reset cascade since v0.23); no vLLM patch and no new dependency (mooncake-transfer-engine is already a hard dep and ships mooncake_master).

Usage

policy:
  generation:
    vllm_cfg:
      env_vars:
        MOONCAKE_CONFIG_PATH: /path/to/mooncake_config.json
    vllm_kwargs:
      kv_transfer_config:
        kv_connector: MooncakeStoreConnector
        kv_role: kv_both

This is unrelated to data_plane.backend=mooncake_cpu (TransferQueue transport for rollout data, not KV cache).

Testing

GPU e2e on GB200 (1 node, 2 GPUs, colocated GRPO, Qwen3-0.6B, 2 steps, sync engine, local mooncake_master):

• Both vLLM replicas launched with KVTransferConfig(kv_connector='MooncakeStoreConnector', kv_role='kv_both') — the override lands as a typed config.
• Reset cascade fired: 8× "Mooncake store reset via remove_all succeeded" in engine logs (scheduler + worker side, both replicas, each step boundary).
• Store actively used: mooncake master log shows ~390 put/batch operations.
• Training correctness unaffected: standard functional metric checks all PASS — max(train/gen_kl_error)=0.00075 < 0.002, probs_ratio_clamped_{min,max} within [0.79, 1.21].
• Baseline behavior without a connector is unchanged (reset-with-no-connector is a no-op success in vLLM ≥0.23; the pinned 0.25.1 applies).

Lint: pre-commit (ruff, ruff-format) and pyrefly check (0 errors) pass on the touched files.

Duplicate-work check

No existing PR or issue in NVIDIA-NeMo/RL touches kv_transfer_config / KV connectors. All current mooncake-related PRs (#2439, #2935, #3501) are TransferQueue data-plane transport, a different layer.

AI assistance disclosure

This PR was authored with AI assistance (Claude). Every changed line was reviewed by @aoshen02, and the e2e validation above was run on real hardware.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[aoshen02]

/ok to test 12617de