Skip to content

Guard preparation off the claim path; pinned registration by default - #116

Merged
mkhazraee merged 1 commit into
ai-dynamo:mainfrom
karya0:fix/guard-async-prepare
Oct 26, 2026
Merged

mkhazraee merged 1 commit into
ai-dynamo:mainfrom
karya0:fix/guard-async-prepare

Conversation

@karya0

@karya0 karya0 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Guard preparation (NIXL init + pool registration) no longer blocks the claim response; promotion waits for it instead, and the Guard core startup budget defaults to 60 s and can be raised with --startup-timeout-seconds (KVCRConfig.startup_timeout_seconds). Cores released during registration are kept and closed once registered; a release never waits. Preparation failure remains service-fatal.

Host pools are registered pinned by default: the quick-start image no longer sets UCX_REG_NONBLOCK_MEM_TYPES, and the dev-guide explains why ODP is not recommended. With ODP, registering a 50 GiB pool failed intermittently with EBUSY under memory compaction; without it, the first registration of a 150 GiB Guard pool takes 12–19 s on an 8-CPU sidecar, more than the claim path could absorb.

@karya0
karya0 force-pushed the fix/guard-async-prepare branch from ba39ad0 to 9c0895c Compare October 21, 2026 23:10
@karya0 karya0 changed the title Large-memory handling: pinned registration by default, async Guard preparation, size-scaled startup timeout Guard preparation off the claim path; pinned registration by default Oct 21, 2026
@karya0
karya0 marked this pull request as ready for review October 21, 2026 23:16
Copilot AI balanced review requested due to automatic review settings October 21, 2026 23:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Deferred cleanup can lose live cores after repeated releases or close failures, allowing pool memory to be unmapped before native resources are released.

2 open issues
What changed in this PR

This PR moves Guard memory registration off the claim path, allowing claims to return while preparation continues and making promotion wait for completion.

Changes:

  • Prepare Guard cores asynchronously and defer cleanup when release occurs during registration.
  • Scale the default startup budget by registered memory size.
  • Remove quick-start ODP settings and document pinned registration behavior.
File Description
tests/​unit/​test_progress.py Tests startup budgets, preparation failures, and close behavior.
tests/​unit/​test_guard.py Tests asynchronous preparation, promotion waits, and deferred release cleanup.
src/​kvcr/​progress.py Adds nonblocking preparation and size-based startup budgets.
src/​kvcr/​guard.py Starts background preparation and tracks deferred core cleanup.
docs/​dev-guide.md Documents startup timing, pinned registration, and ODP limitations.
Dockerfile.quick-start Removes environment settings that enabled ODP.

Review effort: Balanced

Comment thread src/kvcr/guard.py Resolved
Comment thread src/kvcr/guard.py Resolved
Comment thread docs/dev-guide.md Resolved
Comment thread docs/dev-guide.md Resolved
Comment thread src/kvcr/progress.py Resolved
@karya0
karya0 force-pushed the fix/guard-async-prepare branch from 984d8f7 to 95b3cb2 Compare October 24, 2026 00:27
Guard preparation (NIXL init + pool registration) no longer blocks the claim response; promotion waits for it instead, and the Guard core startup budget defaults to 60 s and can be raised with `--startup-timeout-seconds` (`KVCRConfig.startup_timeout_seconds`). Cores released during registration are kept and closed once registered; a release never waits. Preparation failure remains service-fatal.

Host pools are registered pinned by default: the quick-start image no longer sets `UCX_REG_NONBLOCK_MEM_TYPES`, and the dev-guide explains why ODP is not recommended. With ODP, registering a 50 GiB pool failed intermittently with EBUSY under memory compaction; without it, the first registration of a 150 GiB Guard pool takes 12–19 s on an 8-CPU sidecar, more than the claim path could absorb.

Signed-off-by: Kapil Arya <kapila@nvidia.com>
@karya0
karya0 force-pushed the fix/guard-async-prepare branch from 95b3cb2 to 19f8d9e Compare October 26, 2026 17:34
@mkhazraee
mkhazraee merged commit 09a1f4f into ai-dynamo:main Oct 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants