Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
597 changes: 396 additions & 201 deletions csrc/libtorch_stable/hisparse_kernels.cu

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion csrc/libtorch_stable/ops.h
Original file line number Diff line number Diff line change
Expand Up @@ -680,7 +680,7 @@ void hisparse_resolve_residency(
torch::stable::Tensor& device_global_indices,
torch::stable::Tensor& lru_slots,
std::optional<torch::stable::Tensor> const& request_state_indices,
int64_t region_stride,
int64_t region_stride, int64_t max_union_rows,
std::optional<torch::stable::Tensor> const& miss_mask,
std::optional<torch::stable::Tensor> const& stats,
std::optional<torch::stable::Tensor> const& attention_indices,
Expand Down
1 change: 1 addition & 0 deletions csrc/libtorch_stable/torch_bindings.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -995,6 +995,7 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C_cache_ops, ops) {
" Tensor! lru_slots,"
" Tensor? request_state_indices,"
" int region_stride,"
" int max_union_rows,"
" Tensor(a!)? miss_mask=None,"
" Tensor(b!)? stats=None,"
" Tensor(c!)? attention_indices=None,"
Expand Down
8 changes: 4 additions & 4 deletions docs/design/hisparse.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,10 +130,10 @@ path. The resolver consumes the existing graph-stable request mapping from
attention metadata; neither the worker nor individual cache handles keep a
duplicate mapping.

Speculative decoding resolves and consumes each verification step in order.
Each step receives distinct replayable plan rows while sharing the request's
hot-cache state, so a later step cannot reuse a hot row before an earlier step
has consumed it.
Speculative decoding resolves all verification rows of a request in one pass:
one block resolves the union of the rows' top-k against the request's hot-cache
state, so rows that select the same host row share its hot row and no row
evicts a hot row another row of the step still reads.

## P/D import target

Expand Down
Loading
Loading