fix(kv-router): detach dead radix nodes from their parent on removal

[Shang-Pin]

Overview:

RadixTree never frees the memory of removed blocks: apply_removed (and worker
removal / Cleared) clears worker state and lookup entries, but a node whose
workers are all gone is never detached from its parent's children map — nodes
have no parent link, and clear_children_if_unreachable only clears the node's
own children. Every unique stored chain therefore permanently retains its
edge Vec and edge_index hashbrown map as an unreachable-but-alive "zombie"
under the immortal root.

On a production disaggregated frontend serving unique-heavy traffic (~30 rps),
this leaked ~1 GB/h of router memory indefinitely: jeprof showed 9.6 GB of live
allocations after 11 h, 98% under RadixTree::apply_event_with_counters →
RadixBlock::for_blocks, while the tracked-blocks gauge stayed healthily
bounded (350–870k, oscillating with load) — the tree forgets blocks but never
frees their nodes. The arithmetic closes: ~1M requests × ~140 blocks/chain ×
~65 B/block ≈ the measured heap. An A/B with only this fix reverted reproduced
the ~1 GB/h growth; with the fix, the same frontend held a flat ~1 GB working
set over a 3-day soak.

The leak affects every removal path, including the TTL prune (it funnels
through the same apply_removed); kv-events mode is hit hardest because
nothing else bounds the tree. Hot shared prefixes are unaffected (dead nodes
under them get revived by the next store) — it is the unique-suffix tail that
accumulates, so severity scales with prompt uniqueness.

Details:

• Add a Weak parent back-link to RadixBlock (no Rc cycles), wired at all
  node creation/move sites: apply_stored's two child-creation sites, and
  split_node (the new suffix node, plus re-pointing the children it inherits).
• New detach_if_dead: once a node holds no workers and has no children,
  remove it from its parent's children map, cascading upward while ancestors
  become empty; stops at the root (dangling parent). Guarded by Rc::ptr_eq
  against slot reuse.
• Called from the three removal paths: apply_removed,
  remove_or_clear_worker_blocks (also covers Cleared), and
  remove_worker_dp_rank.
• Behavior-preserving otherwise: nodes still alive for another worker (or with
  live children) are untouched; a re-stored chain after detach recreates
  cleanly.
• 5 regression tests: full-chain detach, cascade through split ancestors,
  multi-worker sparing, worker-removal detach, store-after-detach recreation.

Where should the reviewer start?

lib/kv-router/src/indexer/radix_tree.rs — detach_if_dead and its three call
sites; then the parent wiring in split_node (the inherited-children
re-pointing is the one non-obvious spot).

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

Summary by CodeRabbit

• Bug Fixes
  • Removed inactive routing branches after workers are removed, preventing stale cache data from being retained.
  • Improved cleanup of nested and split routing paths.
  • Preserved routing nodes that are still in use by other workers.
  • Ensured removed paths can be recreated correctly when needed.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi Shang-Pin! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 76dd8b77-3c66-465e-8084-1ed94f29cdd0

📥 Commits

Reviewing files that changed from the base of the PR and between 8d43c38 and 270e4ef.

📒 Files selected for processing (1)

• lib/kv-router/src/indexer/radix_tree.rs

---

Walkthrough

The radix tree gains weak parent backlinks and cascading removal of empty nodes. Worker-removal paths invoke detachment, while tests cover split-ancestor cascades, shared-node retention, and chain recreation.

Changes

Radix tree dead-node detachment

Layer / File(s)	Summary
Parent backlink maintenance lib/kv-router/src/indexer/radix_tree.rs	RadixBlock stores weak parent references, which are initialized and updated for created, split, and moved nodes.
Dead-node removal integration lib/kv-router/src/indexer/radix_tree.rs	detach_if_dead removes empty nodes from parent child maps and is called by block and worker removal paths.
Detachment behavior coverage lib/kv-router/src/indexer/radix_tree.rs	Tests verify cascading detachment, shared-worker node retention, and chain recreation after removal.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: detaching dead radix nodes on removal.
Description check	✅ Passed	The description follows the template with Overview, Details, reviewer start, and a completed no-issue linkage section.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[dynamo-ops]

Unsigned Commits Detected

The following commits are not GPG-signed and must be signed before CI can run:

Commit	Reason
8537539	unsigned

Please sign your commits and push again. See the GitHub docs on commit signature verification for help.

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.