Skip to content

[Core] Carry KV transfer, offload and tracing state across process checkpoints - #3

Draft
galletas1712 wants to merge 1 commit into
snapshot-shared-lifecyclefrom
snapshot-kv-connector-checkpoint
Draft

galletas1712 wants to merge 1 commit into
snapshot-shared-lifecyclefrom
snapshot-kv-connector-checkpoint

Conversation

@galletas1712

@galletas1712 galletas1712 commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #1. Lets prefill and decode workers be checkpointed and restored independently, and reconnect.

  • vllm/v1/checkpoint_resources.py: a registry next to the device communicator hooks. Objects that hold state a checkpoint must not contain (network endpoints, NIC registrations, memory shared with other processes) register themselves and implement checkpoint_prepare() / checkpoint_restore(). The engine and each worker run the resources of their own process. New backends plug in without engine changes.
  • A resource that cannot be released once started reports restartable = False. Preparation then fails before anything is released.
  • A restored process gets a new KV engine id (with its _dp{rank} suffix), passed to workers through checkpoint_restore. Replicas restored from one checkpoint stay distinct, and peers re-handshake instead of reusing cached metadata.
  • Waking the KV cache republishes connector handshake metadata before scheduling resumes.
  • supports_checkpoint capability for config validation, alongside supports_sleep_mode. NIXL with sleep mode requires --checkpoint-prepare-on-startup. Connectors that cannot follow a KV cache remap release it only for the startup checkpoint.
  • NIXL pull: UCX caches network state, such as routes keyed by interface index, for the life of the process. An engine that prepares at startup creates its agent only after restore, and KV registration moves to restore_kv_caches. Checkpointing an existing agent is refused.
  • NIXL pull: the side-channel host follows VLLM_NIXL_SIDE_CHANNEL_HOST from the restore environment ([Frontend][Core] Add HTTP checkpoint resume/status lifecycle #1). Without one, the captured value is an interface address of the source pod, so prepare records which interface carries it and restore takes that interface's address in the new pod (same IP family). Names such as DNS names or localhost are kept.
  • CPU offloading: the shared host region is unpinned and its pages freed before capture, then pinned again after restore at the same addresses.
  • OTEL: otlp_traces_endpoint is the first engine argument a restore may change (checkpoint_restore(env, flags) from [Frontend][Core] Add HTTP checkpoint resume/status lifecycle #1). Every process rebuilds its tracer from it and OTEL_RESOURCE_ATTRIBUTES after restore. OpenTelemetry sets the global tracer provider only once, so vLLM installs a provider that swaps the real one behind it.
  • Out of scope: NIXL push mode, the P2P offload tier, and the torch.zeros host buffer fallback.
  • Related upstream work: [KV Connector] Support sleep mode with the NIXL pull connector vllm-project/vllm#59635 (NIXL sleep-mode re-registration) and [KV Connector][NIXL] Add restart lifecycle hooks for checkpoint restore vllm-project/vllm#56852 (NIXL restart hooks).

@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from 6e5d71f to e3bf757 Compare October 14, 2026 21:54
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from b9fca66 to 221a630 Compare October 14, 2026 22:13
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from e3bf757 to 6c8181a Compare October 14, 2026 22:34
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from 221a630 to c5cca04 Compare October 14, 2026 22:34
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from 6c8181a to 2c9edbf Compare October 14, 2026 22:51
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from c5cca04 to 5db9cfe Compare October 14, 2026 22:51
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from 2c9edbf to dd220f0 Compare October 15, 2026 03:49
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from 5db9cfe to 007029b Compare October 15, 2026 03:49
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from dd220f0 to ab5ff6b Compare October 15, 2026 04:00
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from 007029b to 23f9eac Compare October 15, 2026 04:00
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from ab5ff6b to 9bda0a6 Compare October 15, 2026 04:04
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch 2 times, most recently from 17dc9c4 to cc9edbb Compare October 15, 2026 04:05
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch 2 times, most recently from efaff4d to d7184c9 Compare October 15, 2026 04:10
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from cc9edbb to e93c10e Compare October 15, 2026 04:10
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from d7184c9 to 002539b Compare October 15, 2026 04:14
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch 2 times, most recently from 850866c to 5b31960 Compare October 15, 2026 04:23
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from 002539b to 3b17217 Compare October 15, 2026 04:25
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch 2 times, most recently from 9e40fa9 to 319e49f Compare October 15, 2026 04:38
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from 3b17217 to 2310006 Compare October 15, 2026 04:58
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from 319e49f to e52c4b7 Compare October 15, 2026 04:58
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from 2310006 to c3c7adc Compare October 15, 2026 05:42
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from e52c4b7 to e00a945 Compare October 15, 2026 05:42
@galletas1712 galletas1712 mentioned this pull request Oct 15, 2026
1 task done
…eckpoints

Add a checkpoint resource registry next to the device communicator hooks.
Objects that hold state a checkpoint must not contain (network endpoints, NIC
registrations, memory shared with other processes) register themselves and
implement checkpoint_prepare/checkpoint_restore. A resource that cannot be
released once started reports restartable=False, and preparation then fails
before anything is released.

A restored process gets a new KV engine id so peers re-handshake. Waking the
KV cache republishes handshake metadata before scheduling resumes.

The NIXL pull connector supports this when the engine prepares at startup.
UCX caches network state such as routes by interface index for the life of
the process, so the agent is created only after restore. The handshake
listener restarts on the side-channel host from the restore environment, or
on the same interface's address in the new pod. Connectors that cannot
follow a KV cache remap release it only to checkpoint at startup.

The CPU offloading worker unpins and frees its shared host region before
capture and pins it again after restore.

otlp_traces_endpoint becomes the first engine argument a restore may change.
Every process rebuilds its OTEL tracer from it and OTEL_RESOURCE_ATTRIBUTES,
behind a global tracer provider installed once.

Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
@galletas1712
galletas1712 force-pushed the snapshot-shared-lifecycle branch from c3c7adc to 6ff89c4 Compare October 15, 2026 05:55
@galletas1712
galletas1712 force-pushed the snapshot-kv-connector-checkpoint branch from e00a945 to df015f5 Compare October 15, 2026 05:55
@galletas1712 galletas1712 changed the title [KV Connector] Carry NIXL and CPU offload buffers across process checkpoints [Core] Carry KV transfer, offload and tracing state across process checkpoints Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant