Repository navigation
[Core] Carry KV transfer, offload and tracing state across process checkpoints - #3
Draft
galletas1712 wants to merge 1 commit into
Draft
galletas1712 wants to merge 1 commit into
galletas1712 wants to merge 1 commit into
Conversation
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 14, 2026 21:54
6e5d71f to
e3bf757
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 14, 2026 22:13
b9fca66 to
221a630
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 14, 2026 22:34
e3bf757 to
6c8181a
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 14, 2026 22:34
221a630 to
c5cca04
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 14, 2026 22:51
6c8181a to
2c9edbf
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 14, 2026 22:51
c5cca04 to
5db9cfe
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 15, 2026 03:49
2c9edbf to
dd220f0
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 15, 2026 03:49
5db9cfe to
007029b
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 15, 2026 04:00
dd220f0 to
ab5ff6b
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 15, 2026 04:00
007029b to
23f9eac
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 15, 2026 04:04
ab5ff6b to
9bda0a6
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
2 times, most recently
from
October 15, 2026 04:05
17dc9c4 to
cc9edbb
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
2 times, most recently
from
October 15, 2026 04:10
efaff4d to
d7184c9
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 15, 2026 04:10
cc9edbb to
e93c10e
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 15, 2026 04:14
d7184c9 to
002539b
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
2 times, most recently
from
October 15, 2026 04:23
850866c to
5b31960
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 15, 2026 04:25
002539b to
3b17217
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
2 times, most recently
from
October 15, 2026 04:38
9e40fa9 to
319e49f
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 15, 2026 04:58
3b17217 to
2310006
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 15, 2026 04:58
319e49f to
e52c4b7
Compare
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 15, 2026 05:42
2310006 to
c3c7adc
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 15, 2026 05:42
e52c4b7 to
e00a945
Compare
1 task done
…eckpoints Add a checkpoint resource registry next to the device communicator hooks. Objects that hold state a checkpoint must not contain (network endpoints, NIC registrations, memory shared with other processes) register themselves and implement checkpoint_prepare/checkpoint_restore. A resource that cannot be released once started reports restartable=False, and preparation then fails before anything is released. A restored process gets a new KV engine id so peers re-handshake. Waking the KV cache republishes handshake metadata before scheduling resumes. The NIXL pull connector supports this when the engine prepares at startup. UCX caches network state such as routes by interface index for the life of the process, so the agent is created only after restore. The handshake listener restarts on the side-channel host from the restore environment, or on the same interface's address in the new pod. Connectors that cannot follow a KV cache remap release it only to checkpoint at startup. The CPU offloading worker unpins and frees its shared host region before capture and pins it again after restore. otlp_traces_endpoint becomes the first engine argument a restore may change. Every process rebuilds its OTEL tracer from it and OTEL_RESOURCE_ATTRIBUTES, behind a global tracer provider installed once. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
force-pushed
the
snapshot-shared-lifecycle
branch
from
October 15, 2026 05:55
c3c7adc to
6ff89c4
Compare
galletas1712
force-pushed
the
snapshot-kv-connector-checkpoint
branch
from
October 15, 2026 05:55
e00a945 to
df015f5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1. Lets prefill and decode workers be checkpointed and restored independently, and reconnect.
vllm/v1/checkpoint_resources.py: a registry next to the device communicator hooks. Objects that hold state a checkpoint must not contain (network endpoints, NIC registrations, memory shared with other processes) register themselves and implementcheckpoint_prepare()/checkpoint_restore(). The engine and each worker run the resources of their own process. New backends plug in without engine changes.restartable = False. Preparation then fails before anything is released._dp{rank}suffix), passed to workers throughcheckpoint_restore. Replicas restored from one checkpoint stay distinct, and peers re-handshake instead of reusing cached metadata.supports_checkpointcapability for config validation, alongsidesupports_sleep_mode. NIXL with sleep mode requires--checkpoint-prepare-on-startup. Connectors that cannot follow a KV cache remap release it only for the startup checkpoint.restore_kv_caches. Checkpointing an existing agent is refused.VLLM_NIXL_SIDE_CHANNEL_HOSTfrom the restore environment ([Frontend][Core] Add HTTP checkpoint resume/status lifecycle #1). Without one, the captured value is an interface address of the source pod, so prepare records which interface carries it and restore takes that interface's address in the new pod (same IP family). Names such as DNS names orlocalhostare kept.otlp_traces_endpointis the first engine argument a restore may change (checkpoint_restore(env, flags)from [Frontend][Core] Add HTTP checkpoint resume/status lifecycle #1). Every process rebuilds its tracer from it andOTEL_RESOURCE_ATTRIBUTESafter restore. OpenTelemetry sets the global tracer provider only once, so vLLM installs a provider that swaps the real one behind it.torch.zeroshost buffer fallback.