You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
refactor(bench): share the KV-router replay deadline clock - #17887
The active-sequences, approximate-LRU and Mooncake replay benchmarks in lib/bench/kv_router/ each had their own copy of the same deadline clock. It sleeps to an absolute CLOCK_MONOTONIC deadline on Linux, then spins for the last few microseconds. This PR keeps one copy in lib/bench/kv_router/common/clock.rs, and all three benchmarks use it. Timing behavior, the JSON timer name, and feature gating do not change.
Key Changes
The new module holds the existing clock code. The three copies differed only in wording.
It compiles under the same feature gate as the other shared benchmark modules, so the DC CKF benchmarks do not build it.
The clock methods and elapsed_ns now live in the shared module. All three benchmarks use the same code. This PR makes no performance claim.
Search for other copies. A git grep of all tracked files for the clock's system calls, names and spin waits found no copies beyond these three. Similar code was left alone because it works differently: the DC CKF benchmarks' relative sleeps, the mocker's async timerfd timer, CPU-time readers, async condition polls, and spin_loop calls on barriers and locks.
Validation
Initial validation on 1177f1d392 passed on Linux x86_64, including the replay tests for all three benchmarks. The clock unit test was removed after review because its assertions only repeated the trailing spin loop. Non-Linux compilation was not run, and timer precision and EINTR retry were not measured. Subsequent changes are validated through remote CI.
Move the duplicated BenchmarkClock and its Linux absolute CLOCK_MONOTONIC
sleep from the active-sequences, approximate-LRU and Mooncake replay
benchmarks into kv_router/common/clock.rs. Timing behavior is unchanged.
Refs DYN-5211
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
… bench
The shared BenchmarkClock took over the only non-Linux use of Duration in
mooncake_open_loop.rs, so the cfg-gated import is now unused on every
platform.
Refs DYN-5211
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Checks the changed Rust crates with cargo check and Clippy.
Passed (exit 0)
On commit 1177f1d392: workspace and bindings cargo check, feature-gated benchmark builds (including approximate-LRU, which CI does not build), Clippy with -D warnings, the three replay tests and the new clock unit test.
Non-Linux compilation was not exercised: only the Linux target was installed.
Detailed evidence — validation complete
Left out of the table because CI repeats them: none.
Evidence tally: [1/1 validated]
Advisory AI assessment: sound. This is an AI agent's judgment, not an approval.
Validation result: pass — on commit 1177f1d392, every planned format, compile, feature-build, Clippy (-D warnings), and test command exited 0, including the three required replay tests (1 passed each) and the new clock unit test; non-Linux compilation could not be run in this Linux-only environment and is stated as not exercised.
Evidence audit: complete [1/1 validated] — the command report comes from recorded runs.
Commands and results [1/1 validated]
Generated from the commands recorded during this run.
Check 1
Checks the changed Rust crates with cargo check and Clippy.
Result: Passed (exit 0)
Command:
Not shown because the exact command contained private run data.
👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo.
Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.
No actionable comments were generated in the recent review.
Walkthrough
The change adds a shared BenchmarkClock module and updates three open-loop benchmark files to use it instead of their local clock implementations.
Changes
Benchmark clock reuse
Layer / File(s)
Summary
Add and expose the shared clock lib/bench/kv_router/common/clock.rs, lib/bench/kv_router/common/mod.rs
Adds BenchmarkClock with Linux monotonic timing and platform-specific waiting, plus tests for wait behavior. Exposes the module behind the listed feature gates.
Use the shared clock in benchmarks lib/bench/kv_router/active_sequences_open_loop.rs, lib/bench/kv_router/approximate_lru_bench.rs, lib/bench/kv_router/mooncake_open_loop.rs
Imports the shared clock and removes the local clock implementations and Linux sleep helpers from the three benchmark files.
The shared clock is available to the benchmark builds described here. No issue identified in this review prevents merging after normal checks.
Pre-merge checks: 4 · 1
Failed checks: 1 warning
Check name
Status
Explanation
Resolution
Docstring Coverage
Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed checks: 4 passed
Check name
Status
Explanation
Title check
The title clearly and concisely describes the main change: sharing the KV-router replay deadline clock across benchmarks.
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description check
The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the checklist and a closing link to DYN-5211. The Overview states …
Full details: Description check
Explanation
The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the checklist and a closing link to DYN-5211. The Overview states the goal: the three replay benchmarks each held a copy of the same deadline clock, and this PR moves that code into one shared module. Key Changes lists the new module, its feature gate, and the moved clock methods. Validation states what passed on Linux x86_64 and what was not run (non-Linux compilation, timer precision, EINTR retry). The checklist item is ticked. Minor gaps: the 'Closes' line sits outside the Issues and Ongoing Work section as a bullet-less link, and the Key Changes bullet 'The clock methods and elapsed_ns now live in the shared module' does not name elapsed_ns in the raw summary, so that reference is not verifiable from the diff summary. The description is mostly complete and specific.
Full details: Docstring Coverage
Explanation
Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 5 functions across 5 files.
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Warning
Some tools did not complete. Review the errors below.
ast-grep (0.51.1)
ast-grep could not parse rule config: /ast-grep-rules/ast-grep-essentials/rules/javascript/security/detect-child-process.yml
Tip
🎯 Customize your risk assessments
Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.
✅ Automatic approvals, with your rules
Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.
Reviewing files that changed from the base of the PR and between ff072fc and 1177f1d.
Files selected for processing: 5
lib/bench/kv_router/active_sequences_open_loop.rs
lib/bench/kv_router/approximate_lru_bench.rs
lib/bench/kv_router/common/clock.rs
lib/bench/kv_router/common/mod.rs
lib/bench/kv_router/mooncake_open_loop.rs
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.
nursery: Changed nothing for the docstring coverage warning in CodeRabbit’s summary. common/clock.rs already explains the Linux deadline sleep and the portable fallback. The functions keep the existing timing behavior. The repository review guidance in .agents/skills/graham-code-review/SKILL.md asks for minimal comments, and the branch rules do not require this coverage metric.
Second upstream merge for the split v2 PR: 36 commits past the first
merge's base 182039d, 17 conflicted paths, each upstream change placed
in the crate that owns the code at the stack head, upstream behavior whole,
the stack's moves intact, no shims.
- ai-dynamo#17723 flatten policy classes: flat PolicyProfile (from_classes,
UnknownPolicyClassError) in lib/router/admission profile.rs; the YAML
half (default_policy_class, removed keys rejected) in lib/router/kv-router
policy_config.rs; KvSchedulerError::UnknownPolicyClass via From in the
kernel; 400/InvalidArgument in lib/llm (UnknownPolicyClassRefusal) and
ext-proc.
- ai-dynamo#17724 EDF ordering: RouterQueuePolicy::Edf in admission profile.rs;
due-time order, DueKey and class budgets in admission queue.rs; the
restored score keys (RouterSchedulingPolicy::score_policy) as the
kernel's policy.rs (ordered-float back in dynamo-scheduler).
- ai-dynamo#17725 controlled delay: head expiry and with_class_budget_expiry in
admission queue.rs; ClassTimers, due-time checks, DRR charged on
admission (extract_next/settle) and the retained overlap refresh in the
kernel's queue.rs and overlap_refresh.rs; metrics in lib/llm.
- ai-dynamo#17437 image-aware KV routing for TRT-LLM sidecars: zmq_wire/trtllm.rs
in lib/kv-events; the sidecar depends on kv-events, kv-hashing,
multimodal and protocol instead of kv-router.
- ai-dynamo#14775 prefill cancellation, ai-dynamo#17716/ai-dynamo#18077 pool relay, ai-dynamo#17707 NVTX
range, ai-dynamo#18102 LiveEngine buffering, ai-dynamo#17887 shared replay clock: lib/llm,
bindings, mocker, bench, auto-merged with import re-points.
Tests: upstream's queue tests split as the stack does (YAML-driven policy
queue tests in kv-router through testing hooks; kernel tests build the
flat profile through PolicyProfile::from_classes); every upstream test in
the touched modules is present (97 kernel queue, 40 policy queue, 21
policy config, 53 config, 15 policy, 52 zmq_wire). AGENTS.md text for the
queue placed in the two owner crates. CODEOWNERS regenerated; four
lockfiles regenerated and --locked clean.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub Actions
Inactive
github-actions
Bot
added
the
external-contribution
jthomson04
left a comment
Overview
The active-sequences, approximate-LRU and Mooncake replay benchmarks in
lib/bench/kv_router/each had their own copy of the same deadline clock. It sleeps to an absoluteCLOCK_MONOTONICdeadline on Linux, then spins for the last few microseconds. This PR keeps one copy inlib/bench/kv_router/common/clock.rs, and all three benchmarks use it. Timing behavior, the JSONtimername, and feature gating do not change.Key Changes
elapsed_nsnow live in the shared module. All three benchmarks use the same code. This PR makes no performance claim.Search for other copies. A
git grepof all tracked files for the clock's system calls, names and spin waits found no copies beyond these three. Similar code was left alone because it works differently: the DC CKF benchmarks' relative sleeps, the mocker's asynctimerfdtimer, CPU-time readers, async condition polls, andspin_loopcalls on barriers and locks.Validation
Initial validation on
1177f1d392passed on Linux x86_64, including the replay tests for all three benchmarks. The clock unit test was removed after review because its assertions only repeated the trailing spin loop. Non-Linux compilation was not run, and timer precision andEINTRretry were not measured. Subsequent changes are validated through remote CI.Issues and Ongoing Work
None.
Pull Request Checklist
/pr-checklistskill.Closes DYN-5211.
Summary by CodeRabbit