Repository navigation
Guard preparation off the claim path; pinned registration by default - #116
Merged
Merged
Conversation
karya0
force-pushed
the
fix/guard-async-prepare
branch
from
October 21, 2026 23:10
ba39ad0 to
9c0895c
Compare
karya0
marked this pull request as ready for review
October 21, 2026 23:16
There was a problem hiding this comment.
🟡 Changes recommended
Deferred cleanup can lose live cores after repeated releases or close failures, allowing pool memory to be unmapped before native resources are released.
2 open issues
What changed in this PR
This PR moves Guard memory registration off the claim path, allowing claims to return while preparation continues and making promotion wait for completion.
Changes:
- Prepare Guard cores asynchronously and defer cleanup when release occurs during registration.
- Scale the default startup budget by registered memory size.
- Remove quick-start ODP settings and document pinned registration behavior.
| File | Description |
|---|---|
tests/unit/test_progress.py |
Tests startup budgets, preparation failures, and close behavior. |
tests/unit/test_guard.py |
Tests asynchronous preparation, promotion waits, and deferred release cleanup. |
src/kvcr/progress.py |
Adds nonblocking preparation and size-based startup budgets. |
src/kvcr/guard.py |
Starts background preparation and tracks deferred core cleanup. |
docs/dev-guide.md |
Documents startup timing, pinned registration, and ODP limitations. |
Dockerfile.quick-start |
Removes environment settings that enabled ODP. |
Review effort: Balanced
mkhazraee
reviewed
Oct 23, 2026
mkhazraee
reviewed
Oct 23, 2026
mkhazraee
reviewed
Oct 23, 2026
karya0
force-pushed
the
fix/guard-async-prepare
branch
from
October 24, 2026 00:27
984d8f7 to
95b3cb2
Compare
mkhazraee
approved these changes
Oct 24, 2026
This was referenced Oct 26, 2026
Guard preparation (NIXL init + pool registration) no longer blocks the claim response; promotion waits for it instead, and the Guard core startup budget defaults to 60 s and can be raised with `--startup-timeout-seconds` (`KVCRConfig.startup_timeout_seconds`). Cores released during registration are kept and closed once registered; a release never waits. Preparation failure remains service-fatal. Host pools are registered pinned by default: the quick-start image no longer sets `UCX_REG_NONBLOCK_MEM_TYPES`, and the dev-guide explains why ODP is not recommended. With ODP, registering a 50 GiB pool failed intermittently with EBUSY under memory compaction; without it, the first registration of a 150 GiB Guard pool takes 12–19 s on an 8-CPU sidecar, more than the claim path could absorb. Signed-off-by: Kapil Arya <kapila@nvidia.com>
karya0
force-pushed
the
fix/guard-async-prepare
branch
from
October 26, 2026 17:34
95b3cb2 to
19f8d9e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Guard preparation (NIXL init + pool registration) no longer blocks the claim response; promotion waits for it instead, and the Guard core startup budget defaults to 60 s and can be raised with
--startup-timeout-seconds(KVCRConfig.startup_timeout_seconds). Cores released during registration are kept and closed once registered; a release never waits. Preparation failure remains service-fatal.Host pools are registered pinned by default: the quick-start image no longer sets
UCX_REG_NONBLOCK_MEM_TYPES, and the dev-guide explains why ODP is not recommended. With ODP, registering a 50 GiB pool failed intermittently with EBUSY under memory compaction; without it, the first registration of a 150 GiB Guard pool takes 12–19 s on an 8-CPU sidecar, more than the claim path could absorb.