fix(snapshot): exit vLLM source without engine teardown - #12184
Merged
Conversation
|
🎯 Code Coverage (details) 🔗 Commit SHA: 58272b1 | Docs | Datadog PR Page | Give us feedback! |
galletas1712
force-pushed
the
fix/vllm-snapshot-source-exit
branch
from
July 27, 2026 21:52
3abe47b to
0e09c2e
Compare
cursor
Bot
force-pushed
the
fix/vllm-snapshot-source-exit
branch
from
July 27, 2026 21:54
0e09c2e to
15a1614
Compare
cursor
Bot
force-pushed
the
fix/vllm-snapshot-source-exit
branch
from
July 27, 2026 21:59
15a1614 to
6207d3b
Compare
galletas1712
marked this pull request as ready for review
July 27, 2026 22:00
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
WalkthroughThe vLLM snapshot success path now logs completion and terminates the process with ChangesSnapshot restore termination
Estimated code review effort: 2 (Simple) | ~5 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
Contributor
Author
|
/ok to test 6207d3b |
julienmancuso
approved these changes
Jul 27, 2026
karen-sy
approved these changes
Jul 27, 2026
galletas1712
enabled auto-merge (squash)
July 27, 2026 23:27
galletas1712
force-pushed
the
fix/vllm-snapshot-source-exit
branch
from
July 28, 2026 03:48
6207d3b to
f02b4cb
Compare
Contributor
Author
|
/ok to test f02b4cb |
galletas1712
force-pushed
the
fix/vllm-snapshot-source-exit
branch
2 times, most recently
from
July 28, 2026 21:12
4232724 to
7b55ac9
Compare
Contributor
Author
|
/ok to test 7b55ac9 |
galletas1712
force-pushed
the
fix/vllm-snapshot-source-exit
branch
from
July 29, 2026 20:02
7b55ac9 to
d8a06c3
Compare
Contributor
Author
|
/ok to test d8a06c3 |
galletas1712
force-pushed
the
fix/vllm-snapshot-source-exit
branch
from
July 30, 2026 00:45
d8a06c3 to
b1f88b2
Compare
Contributor
Author
|
/ok to test b1f88b2 |
2 tasks
galletas1712
force-pushed
the
fix/vllm-snapshot-source-exit
branch
from
July 30, 2026 06:43
b1f88b2 to
afc98d1
Compare
Contributor
Author
|
/ok to test afc98d1 |
After a successful initial capture, exit the source process immediately with os._exit so engine subprocess teardown cannot keep the checkpoint Job alive or SIGSEGV during CUDA cleanup. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
Contributor
Author
|
/ok to test 58272b1 |
galletas1712
force-pushed
the
fix/vllm-snapshot-source-exit
branch
from
July 30, 2026 16:51
afc98d1 to
58272b1
Compare
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Snapshot capture;
shutdown; and
Motivation
This is a generic Dynamo Snapshot/vLLM lifecycle fix, independent of GMS V1.
The initial source path raises
SystemExitafter successful capture. Thatunwinds vLLM and may enter CUDA and EngineCore teardown after the process image
has already been checkpointed. In the observed failure, teardown produced a
source-process SIGSEGV even though capture itself had succeeded.
The successful initial-capture boundary does not need Python cleanup: the
source container is deliberately being replaced by a restore-standby
container. This change therefore exits directly with
os._exit(0), matchingthe existing TensorRT-LLM Snapshot lifecycle. The restored path still returns
normally.
This fix is split from #12011 because it applies to Dynamo Snapshot with or
without GMS.
Validation
Focused test
The focused unit test covers both lifecycle branches:
nscale Snapshot E2E
The fix passed in the final graph-enabled Qwen3-0.6B TP1/DP1 Snapshot + GMS V1
proof on
nscale-dev, nodes2877.The source engine logged:
Its terminal state was:
The replacement entered restore standby, Snapshot reported
nvidia.com/snapshot-restore-status.engine=completed, restore completed in2.674 seconds, and the restored engine served five coherent requests with
cached_tokens=0.The previous destructor-path SIGSEGV did not recur. No CUDA fault, illegal
memory access, late free, engine crash, or restore retry was observed.
Qualification build:
https://github.com/ai-dynamo/dynamo/actions/runs/30227329178.
Summary by CodeRabbit