Repository navigation
sampling: write master rows with a scalar fill_, not through a reused pinned row - #222
Merged
NSagan271 merged 2 commits intoSep 10, 2026
Conversation
vasilevklart
changed the title
vasilevklart
pushed a commit
to vasilevklart/mstar
that referenced
this pull request
Aug 21, 2026
Conflicts: pyproject.toml (keep both per-file ignores), mstar/utils/sampling.py (main's mstar-project#201 multi-sampler file + the staging fence hunks, = mstar-project#222), mstar/engine/cuda_graph_runner.py (piecewise replay: keep the lane's swap/plan/replay try-finally, add mstar-project#201's sampler gather before replay and scatter_offsets after).
vasilevklart
force-pushed
the
users/kirill/pinned-staging-fence
branch
from
September 9, 2026 00:03
9e55017 to
5f0b7c8
Compare
vasilevklart
changed the title
NSagan271
self-requested a review
September 9, 2026 17:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Buffer.write_master_rowstages the value in one pinned[1]row per Buffer and queues anon_blockingH2D from it. The copy engine reads that row only when the stream reaches the copy, i.e. behind whatever the GPU is still running (step N−1 under the 2-step launch bound), while the host has already moved on to the nextwrite_master_rowon the same row._stage_slot_idx→_init_slot→_write_master_rowdoes five of those per newly registered rid, so two rids admitted in the same step (~50 µs apart) land the second rid's temperature / top_k / top_p / seed / repetition penalty in both slots. Same forupdate_request_config. Double-buffering doesn't reach this row: it's per Buffer, not per step or per slot.Fix:
master[slot:slot+1].fill_(value). The scalar is baked into the launch, nothing on the host outlives the call, nothing to fence. Five tiny kernels per registration instead of five DMAs, same stream order ahead of the gather that reads them.row_cpuand the pinned flag onBuffer.allocatego away.Reproduced on an H100 with main's
write_master_rowverbatim and ~20 ms of kernels queued ahead: 20/20 trials bad; 0/20 withfill_(and 0/20 with the event fence this PR previously carried).test/modular/test_sampler_master_row_reuse.pyis that reproducer against the realBuffer(GPU-only, skips on CPU).Dropped from the first version: the fence on the per-step slot-index row. With 2 slots and postprocess(N) syncing N's completion event before N+2 is built, that row can't be refilled while N's DMA is pending on the batched path, and #243 extends the same guarantee to the piecewise runner.