Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/features/nixl_connector_usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ Additional configuration options in `kv_connector_extra_config`:
| `bidirectional_kv_xfer` | `false` | Enable bidirectional D→P KV transfer. |
| `kv_recompute_threshold` | `64` | Minimum number of remote tokens required to trigger a D→P pull. Below this threshold, P recomputes locally instead of pulling (to amortize transfer latency). |
| `decoder_kv_blocks_ttl` | `480` | TTL (seconds) for KV blocks cached on D for bidirectional reuse. Blocks are released after this duration. Not renewed via heartbeats. |
| `whole_row_regions` | `false` | With a block-major (`BLHNC`) layout, register every replicated (MLA) layer of a block row as one NIXL region, so a block transfers as a single descriptor instead of one per layer. Head-sharded layers, such as a GQA draft under an MLA target, keep per-layer regions so each TP rank reads its head slice. Requires device-resident caches on both peers (incompatible with `kv_buffer_device=cpu` and HiSparse host-resident groups) and must be set on both P and D. |

### Multi-turn proxy setup

Expand Down
Loading
Loading