IMA fix by making the copy of book keeping buffer to GPU blocking - #5715
Merged
Conversation
svcnvidia-nemo-ci
marked this pull request as draft
July 8, 2026 19:46
Contributor
|
This PR has been automatically converted to draft because all PRs must start as drafts. When you are ready for review, click Ready for Review to begin the review process. This will:
See the contribution guide for more details. |
Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
Contributor
|
@tdene can you signoff of these changes? |
tdene
approved these changes
Jul 9, 2026
Contributor
|
@lmcafee-nvidia can you please take a look at these changes? |
The pinned `_cpu_bookkeeping_buf` is re-staged in place on the next step, so a non_blocking H2D copy races with host writes and can corrupt token/block indices on the GPU. Use a blocking copy instead; the per-step sync cost is negligible relative to the forward pass. Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
santhnm2
approved these changes
Jul 13, 2026
santhnm2
left a comment
santhnm2
left a comment
Contributor
There was a problem hiding this comment.
Can you update the PR title and description?
shanmugamr1992
changed the title
cuichenx
approved these changes
Jul 14, 2026
Update transfer_bookkeeping_to_gpu docs to match non_blocking=False and explain the host restage race that requires a blocking copy. Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
lmcafee-nvidia
approved these changes
Jul 14, 2026
shanmugamr1992
enabled auto-merge
July 14, 2026 18:51
Contributor
Author
|
/ok to test 75dd58f |
svcnvidia-nemo-ci
added
Approved
sidsingh-nvidia
pushed a commit
to sidsingh-nvidia/Megatron-LM
that referenced
this pull request
Jul 14, 2026
The pinned `_cpu_bookkeeping_buf` is re-staged in place on the next step, so a non_blocking H2D copy races with host writes and can corrupt token/block indices on the GPU. Use a blocking copy instead; the per-step sync cost is negligible relative to the forward pass. Cherry-picked from NVIDIA/Megatron-LM PR NVIDIA#5715. Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
Contributor
Author
|
/ok to test 8cee542 |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/29458845201 |
chochowski
pushed a commit
to chochowski/Megatron-LM
that referenced
this pull request
Jul 20, 2026
…IDIA#5715) Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com> Signed-off-by: mchochowski <mchochowski@nvidia.com>
terminator123
pushed a commit
to 021ai/Megatron-LM
that referenced
this pull request
Aug 3, 2026
…IDIA#5715) Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
svcnvidia-nemo-ci
pushed a commit
to dimapihtar/Megatron-LM
that referenced
this pull request
Aug 4, 2026
…IDIA#5715) Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The original issue. Each inference step, Megatron copies a pinned CPU
bookkeeping buffer (
_cpu_bookkeeping_buf) to the GPU with an async(
non_blocking=True) copy, then immediately reuses/overwrites that same CPUbuffer on the next step. The host doesn't wait for the copy to finish, so it
clobbers bytes that the in-flight GPU copy is still reading. The GPU ends up
with corrupted token/KV-block indices and dereferences out-of-bounds memory
→
illegal memory access. CUDA-graph warmup runs steps back-to-back withalmost no host work in between, so the race fires every time there.
Fix:
to make the copy blocking would make sure this issue doesn't happen
timer aggregates ~370 decode iterations/seq; the per-iteration copy is a few µs and any blocking host-stall (~10–30 µs) is a fraction of a percent of the ~32 ms/iter forward pass — diluted below noise.
With larger models this is even less a concern,