You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(kv-router): append decode stores at the leaf tail under one lock - #18108
Most KV events the router applies are one-block decode stores whose parent is the tail of the worker's own leaf: about 80% of events in the selection replay's agentx workload and 87% in its chat workload. Each went through parent planning: an EdgeIndex probe for the parent, then four more shape-gate round trips before try_extend_leaf_with_version appended. Now, when the worker's lookup entry for the parent names a leaf that ends with the parent and the worker covers that tail, the store is checked and appended under one exclusive shape gate, and the new lookup entries reuse the lane's node id. Every other store takes the existing path unchanged. Removal run grouping also prefetches lookup slots ahead, as upserts and deletes already do.
base
this PR
Selection replay, agentx at saturation: index cycles per applied event
12,672
9,874 (−22%)
Events applied/s (4 event threads)
957k
1,134k (+18%)
Index backlog at window end
4.83M events
97k events
Selection replay, chat at saturation: index cycles per applied event
13,352
11,740 (−12%)
mooncake_bench write capacity
88.5 M ops/s
+1.6% [−1.7, +3.4]
mooncake_bench keep-up find_matches and update p50/p99
inside noise
Tyche Grace node. Selection replay: selection_replay_bench (#17959) at saturation, agentx 3 alternating runs per arm and chat 1 run per arm, with cycles from window-bracketed profiles. mooncake_bench: 6 paired rounds, glibc, 95% bootstrap CIs.
Both arms run at about 12.1k req/s, the scheduler actor's ceiling. With this change the 4 event threads nearly keep up at that rate, instead of falling millions of events behind.
Key Changes
Node::try_append_at_tail: the decode-extension checks (not internal, not an anchor, tail is the parent, worker covers the tail) and the append, under one exclusive gate.
apply_stored tries it first, through the lane's entry for the parent (LaneLookup::held, borrowed, no Arc clone), and points the new entries at that node id (LaneLookup::upsert_all_held).
BlockLookup::leading_holding replaces the unprefetched get loop in LaneLookup::leading_naming.
README (Decode Extension) and tests: the anchor and uncovered-tail parents the one-lock path must decline (no existing test covers those guards), and the existing decode-extension test now asserts the stores take that path.
This adds a commit path that relies on the exclusive gate instead of a shape version. The leaf-extension conditions, including sticky internal, are unchanged.
Validation
cargo test -p dynamo-kv-router --lib
cargo clippy -p dynamo-kv-router --all-targets -- -D warnings, with and without --features bench
Most KV events are one-block decode stores whose parent is the tail of the
worker's own leaf. Each went through parent planning: an EdgeIndex probe for
the parent, then four more shape-gate round trips before the append, an Arc
clone, and a node-address lookup for the new entries. When the worker's
lookup entry for the parent names a leaf that ends with it and the worker
covers that tail, append there under one exclusive gate and point the new
entries at the held node id. Every other store takes the general path
unchanged.
Removal run grouping now prefetches lookup slots ahead, as upserts and
deletes already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No actionable comments were generated in the recent review.
Walkthrough
Stores now attempt to append blocks directly to an eligible compressed leaf tail when the worker’s lookup holds the parent. The change adds lookup support for this path, preserves the existing insertion flow when it does not apply, and adds tests for success and fallback cases.
Changes
Tail append store path
Layer / File(s)
Summary
Lookup access and prefix operations lib/kv-router/src/indexer/concurrent_radix_tree_compressed/block_lookup.rs, lib/kv-router/src/indexer/concurrent_radix_tree_compressed/lane_lookup.rs
BlockLookup::leading_holding counts the matching prefix. LaneLookup::held returns a node ID and borrowed node reference, and upsert_all_held updates entries using that ID. Randomized tests exercise the new lookup operations.
Store tail-append path lib/kv-router/src/indexer/concurrent_radix_tree_compressed/node.rs, lib/kv-router/src/indexer/concurrent_radix_tree_compressed/store.rs, lib/kv-router/src/indexer/concurrent_radix_tree_compressed/mod.rs, lib/kv-router/src/indexer/concurrent_radix_tree_compressed/tests.rs, lib/kv-router/src/indexer/concurrent_radix_tree_compressed/README.md
apply_stored tries to append blocks to a held parent’s leaf tail. Successful appends update lookup entries and lifecycle hashes. Anchors, internal nodes, uncovered tails, and stale entries use the existing insertion flow. Tests cover these cases, and the README describes the lookup-first path.
No actionable merge-blocking risk was found in the tail-append change.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title clearly and concisely describes the primary change: optimizing decode stores by appending at the leaf tail under one lock.
Docstring Coverage
Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentati…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description check
The description follows the template closely. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the checklist. The Overview gives the problem, the approach, and a be…
Full details: Docstring Coverage
Explanation
Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 10 functions across 6 files. (1 skipped: 1 unsupported.)
Full details: Description check
Explanation
The description follows the template closely. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the checklist. The Overview gives the problem, the approach, and a benchmark table with cycles, throughput, backlog, and write capacity figures. Key Changes lists each touched function and the test and README updates. Validation lists the cargo test and clippy commands with and without the bench feature. The only gap is the Issues section. It says 'Found with the selection replay benchmark in #17959' but does not contain a 'Closes #XXXX' issue link. The template marks that link as advisory today. The pull request checklist item is unchecked, and the template expects the author to run the /pr-checklist skill. The description is detailed and on-topic. Its missing elements are minor and non-critical.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.
Main's tests already fail when the internal or tail-hash guard is removed,
or when removal grouping miscounts. Keep the anchor and uncovered-tail
tests, which nothing else covers, and assert the fast path is taken in the
existing decode-extension test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fourth upstream merge for the split v2 PR: the five commits past
02331f0. Four conflicts, all from ai-dynamo#18009, placed in the owner crates;
upstream behavior whole, the stack's moves intact, no shims.
- ai-dynamo#18009 keep prepared rows when ranks are past the reject thresholds:
is_worker_full(dp_size)/skips_full_workers in the kernel's filter.rs;
the actor keeps prepared rows and the kernel's select_with_policy drops
the rows of workers whose every rank is full before the policy adopts
them (PreparedRows::retain in worker-selection; the policy recounts the
best cached prefix from the rows it adopts and reports whether any is
left); prepared rows carry dp_size (u16, so the 128-byte row holds) set
in the kernel's CandidatePreparerExt::prepare; the three new kernel
queue tests and the relocated test helpers updated.
- ai-dynamo#18108 CRTC decode stores appended at the leaf tail under one lock:
lib/router/kv-indexer concurrent_radix_tree_compressed (auto-merged).
- ai-dynamo#16258 vLLM backend defaults for omitted token limits: lib/llm
preprocessor, lib/sidecar/vllm, components, docs, tests (auto-merged).
- ai-dynamo#18152 planner ARIMA forecasts, ai-dynamo#16426 operator crash-loop: outside the
router (auto-merged).
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
the
documentation
Overview
Most KV events the router applies are one-block decode stores whose parent is the tail of the worker's own leaf: about 80% of events in the selection replay's agentx workload and 87% in its chat workload. Each went through parent planning: an
EdgeIndexprobe for the parent, then four more shape-gate round trips beforetry_extend_leaf_with_versionappended. Now, when the worker's lookup entry for the parent names a leaf that ends with the parent and the worker covers that tail, the store is checked and appended under one exclusive shape gate, and the new lookup entries reuse the lane's node id. Every other store takes the existing path unchanged. Removal run grouping also prefetches lookup slots ahead, as upserts and deletes already do.mooncake_benchwrite capacitymooncake_benchkeep-upfind_matchesand update p50/p99Tyche Grace node. Selection replay:
selection_replay_bench(#17959) at saturation, agentx 3 alternating runs per arm and chat 1 run per arm, with cycles from window-bracketed profiles.mooncake_bench: 6 paired rounds, glibc, 95% bootstrap CIs.Both arms run at about 12.1k req/s, the scheduler actor's ceiling. With this change the 4 event threads nearly keep up at that rate, instead of falling millions of events behind.
Key Changes
Node::try_append_at_tail: the decode-extension checks (not internal, not an anchor, tail is the parent, worker covers the tail) and the append, under one exclusive gate.apply_storedtries it first, through the lane's entry for the parent (LaneLookup::held, borrowed, noArcclone), and points the new entries at that node id (LaneLookup::upsert_all_held).BlockLookup::leading_holdingreplaces the unprefetchedgetloop inLaneLookup::leading_naming.This adds a commit path that relies on the exclusive gate instead of a shape version. The leaf-extension conditions, including sticky
internal, are unchanged.Validation
cargo test -p dynamo-kv-router --libcargo clippy -p dynamo-kv-router --all-targets -- -D warnings, with and without--features benchIssues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.🤖 Generated with Claude Code
Summary by CodeRabbit
Performance
Bug Fixes