fix(runtime): give event publishers unique discovery identities - #11014
Merged
Conversation
zhongdaor-nv
temporarily deployed
to
external_collaborator
June 27, 2026 04:49 — with
GitHub Actions
Inactive
zhongdaor-nv
temporarily deployed
to
external_collaborator
June 27, 2026 05:22 — with
GitHub Actions
Inactive
zhongdaor-nv
marked this pull request as ready for review
June 27, 2026 09:22
Contributor
WalkthroughThe PR splits discovery ownership from event-channel publisher identity, threads the new publisher id through event publisher registration and ZMQ subscriber matching, updates discovery logging, adds shutdown-safe unregistration, expands ZMQ integration coverage, and removes a router mocker environment override. ChangesDiscovery and event-plane identity flow
Router mocker harness
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
Comment |
zhongdaor-nv
force-pushed
the
fix/event-publisher-identity
branch
from
June 30, 2026 01:36
21dfb28 to
a75a210
Compare
biswapanda
reviewed
Jul 2, 2026
biswapanda
approved these changes
Jul 3, 2026
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Dropping an EventPublisher schedules an async discovery unregister and holds a GracefulShutdownTracker guard so Runtime::shutdown Phase 2 waits for it. Add a regression test that drops a publisher, runs the real shutdown sequence, and asserts the guard is taken synchronously on drop, released once unregister finishes, and that the publisher is gone from discovery before Phase 3 cancels the main token. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zhongdaor-nv
force-pushed
the
fix/event-publisher-identity
branch
from
July 3, 2026 06:02
f5537c8 to
3a977ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EventPublishera fork-safe, per-incarnation publisher ID and use the same ID in bothEventEnvelopeand event-channel discovery.DiscoveryInstance::EventChannelshape while allowing multiple publishers for the same process/component/topic.DiscoveryInstanceIdand ignore unrelated topics.default_instance_id, registration code logs only the finalinstance_id, and EventPublisher logs usepublisher_id. The old publicwith_instance_idmethod remains as a compatibility alias.The root cause was that each publisher bound its own ZMQ endpoint and owned its own sequence counter, but all publishers in one process registered with the process-level discovery ID. Their discovery keys therefore collided, subscribers only connected to one endpoint, and dropping any colliding publisher could remove the shared registration.
This also makes
(publisher_id, sequence)valid for broker deduplication across multiple publishers and prevents a late unregister from an old publisher incarnation from removing its replacement. The wire schema and discovery path structure are unchanged. The public RustDiscoverySpec::EventChannelvariant now requirespublisher_id.Validation
origin/mainat2469510c7d6and resolved the event-plane conflict while retaining both main's cached NATS subject path and this change's publisher lifecycle handling.cargo fmt --allgit diff --checkcargo test -p dynamo-runtime transports::event_plane:: --lib— 18 passedcargo test -p dynamo-runtime --lib— 433 passed, 2 ignoredcargo clippy -p dynamo-runtime --lib --tests -- -D warningscargo check -p dynamo-llm -p dynamo-backend-commonmaturin develop --uvpytest -q -s 'tests/router/test_router_e2e_with_mockers.py::test_router_decisions[nats_core-tcp]'— 1 passed in 40.75s with the default ZMQ event plane, two workers, anddp_size=4pytest -q -s 'tests/router/test_router_e2e_with_mockers.py::test_router_decisions_disagg_round_robin_prefill_dp_rank[no_bootstrap-prefill_first]'— 1 passed in 80.35s with the default ZMQ event planepytest -q -s 'tests/router/test_router_e2e_with_mockers.py::test_router_decisions[jetstream-tcp]'— 1 passed in 40.64s, confirming durable coverage still opts into NATSpre-commit run --fileson all changed Rust and Python filesSummary by CodeRabbit
New Features
Bug Fixes