fix(mocker): stabilize replay KVBM ordering [DYN-3850]

[PeaBrane]

Summary

• add an opt-in ordered_transfer_starts pipeline mode that preserves batch-input order through synchronous transfer reservation/publication
• enable that mode, plus stable G2 router-event publication ordering, only for replay-bench
• preserve transfer concurrency by releasing each successor immediately after the current transfer has reserved its notification and entered the in-flight phase
• forward replay-bench through the Python binding crate for high-level replay builds

Production behavior is unchanged by default: ordered_transfer_starts defaults to false, and the mocker enables it only behind the existing replay feature.

Root cause

An intermittent vLLM-disaggregated replay mismatch shifted a subset of decode-admit, destination-activation, and terminal timestamps by exact 0.123076 ms increments. Event-level instrumentation showed that Tokio could schedule same-batch G1→G2 tasks in different reservation orders (for example 32,32,32,29 versus 32,29,32,32). That changed which block group completed first and, independently, same-timestamp lifecycle publication could assign router-event IDs in executor order.

The fix orders only the synchronous reservation start and replay router-event publication. It does not serialize transfer completion or the data path.

Validation

• cargo fmt --all -- --check
• cargo test -p kvbm-engine --no-default-features ordered_transfer_starts_ignore_task_schedule_order
• cargo test -p dynamo-mocker --features replay-bench,kvbm-offload g1_to_g2_completion_feeds_g3_with_presence_policy
• requested clean lint/format matrix passed for lib/kvbm-engine, lib/mocker, and lib/bindings/python
• adversarial fresh-state alternating-process soak, small offload batches: baseline 8/8 and fixed build 8/8 produced one shared digest (3bbd0fe36efb8ddfe1af7fceae1c6ab3357fcf9d5aebb3c5b1ae2a2a4cebb98b)
• 3072-block control: baseline 8/8 and fixed build 8/8 produced one shared digest (9f7c4c2f027b30df6b95e14db6d6603f904284625849eb68c35e730f7e23ae90)
• authoritative 5,000-row vLLM-disaggregated replay: both baseline reps and both fixed reps matched byte-for-byte (f0fef647a549713ec261fdc4ff558478caaf78001042f1c5bb933b8f7774f40e)

---

Summary by CodeRabbit

• New Features
  • Added an optional mode that starts data transfers in a predictable input order.
  • Added replay-benchmark support for consistent request sequencing and event ordering.
• Bug Fixes
  • Improved consistency of replay results by ensuring collected events follow their expected sequence.
• Tests
  • Added coverage verifying that transfer operations begin in the configured order, regardless of scheduling.

[github-actions]

👋 Hi PeaBrane! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3b8f3fa9-465a-475e-ae48-c72d1bc3a443

📥 Commits

Reviewing files that changed from the base of the PR and between f62f28e and c4a7067.

📒 Files selected for processing (3)

• lib/bindings/python/Cargo.toml
• lib/kvbm-engine/src/offload/pipeline.rs
• lib/mocker/src/kvbm_offload/engine.rs

---

Walkthrough

Changes

The pipeline gains an optional ordered transfer-start mechanism using chained asynchronous signals. Replay-bench builds enable this behavior and sort G2 router events by sequence hash for deterministic ordering. A test verifies transfer starts follow batch order despite task scheduling.

Replay determinism

Layer / File(s)	Summary
Pipeline ordering configuration lib/kvbm-engine/src/offload/pipeline.rs	Adds the ordered_transfer_starts configuration field, builder method, default, and executor wiring.
Ordered transfer-start coordination lib/kvbm-engine/src/offload/pipeline.rs	Chains transfer tasks with oneshot signals, releases successors at transfer start or skip, and tests batch-order execution.
Replay benchmark integration lib/bindings/python/Cargo.toml, lib/mocker/src/kvbm_offload/engine.rs	Adds the Python replay-bench feature, enables ordered G1-to-G2 starts, and sorts router events by seq_hash.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is detailed, but it misses the required template sections, especially Where should the reviewer start? and Related Issues.	Add the required Overview, Details, Where should the reviewer start?, and Related Issues sections, including the issue link or confirmed no-issue checkbox.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the PR’s main change: stabilizing replay KVBM ordering in mocker.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 41.24% (-3.59%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ae41f05 | Docs | Datadog PR Page | Give us feedback!}

[PeaBrane]

/ok test c4a7067

[PeaBrane]

/ok to test c4a7067

[dynamo-ops]

/ok to test f40bc61

[dynamo-ops]

/ok to test ae41f05