Skip to content

feat(runtime): default event plane to ZMQ for all backends - #10902

Merged
zhongdaor-nv merged 13 commits into
mainfrom
zhongdaor/event-plane-default-zmq
Jun 26, 2026
Merged

zhongdaor-nv merged 13 commits into
mainfrom
zhongdaor/event-plane-default-zmq

Conversation

@zhongdaor-nv

@zhongdaor-nv zhongdaor-nv commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Make ZMQ the default event plane for every discovery backend. Previously the default was backend-driven — local backends (file/mem) used ZMQ while distributed backends (etcd/kubernetes) used NATS. Now ZMQ is the default everywhere; NATS remains a first-class opt-in via DYN_EVENT_PLANE=nats.

Since the request plane already defaults to TCP, a standard etcd deployment now opens no NATS connection by default — NATS is only connected when explicitly requested (DYN_EVENT_PLANE=nats, NATS_SERVER set, DYN_REQUEST_PLANE=nats, durable KV events, or the NATS audit sink).

Changes

  • discovery::EventTransportKind: #[default] and from_env() / from_env_or_default() now resolve to Zmq when DYN_EVENT_PLANE is unset/empty.
  • DiscoveryBackend::resolve_event_transport_kind (authoritative resolver): unset/empty/invalid → Zmq unconditionally (dropped the backend-driven branch).
  • Durable KV events stay NATS-only. The subscriber still refuses to start the JetStream path under a non-NATS transport, but now with an actionable message: set DYN_EVENT_PLANE=nats or drop --durable-kv-events.
  • Tests: tests/conftest.py pins DYN_EVENT_PLANE=nats when durable_kv_events=True so the durable/JetStream path keeps working under the new default.
  • Docs / CLI help: event-plane design doc, router guide, CLAUDE.md, and the --event-plane help text updated to describe ZMQ-default / NATS-opt-in.

Compatibility / blast radius

  • Audited every default-path NATS consumer (audit sink, trace, FPM, LoRA, worker-query recovery, runtime accessors, discovery): all either ride the transport-agnostic event plane, are explicit opt-ins, or handle an absent NATS client gracefully. Nothing in the default path crashes for lack of NATS.
  • Rollback is a single env var: DYN_EVENT_PLANE=nats restores prior behavior.

Testing

  • cargo test --locked -p dynamo-runtime --lib — 395 passed (incl. ZMQ transport tests)
  • cargo clippy --locked --no-deps -p dynamo-runtime -p dynamo-llm — clean
  • Build of dynamo-runtime + dynamo-llm — green

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Configuration Changes

    • Event plane now defaults to ZMQ instead of NATS across all discovery backends.
    • NATS event transport is now opt-in via configuration flag.
  • Documentation & Help Text

    • Updated CLI help text and configuration documentation to reflect new event plane defaults.
    • Improved error messages to guide users on configuring NATS when needed.

Previously the event-plane default was backend-driven: local backends
(file/mem) defaulted to ZMQ while distributed backends (etcd/kubernetes)
defaulted to NATS. Make ZMQ the default for every discovery backend; NATS
remains an explicit opt-in via DYN_EVENT_PLANE=nats. Since the request
plane already defaults to TCP, a standard etcd deployment now needs no
NATS connection by default.

- discovery::EventTransportKind: #[default] and from_env*() -> Zmq
- DiscoveryBackend::resolve_event_transport_kind: unset/empty/invalid -> Zmq
- durable KV events stay NATS-only; the subscriber now bails with an
  actionable message (set DYN_EVENT_PLANE=nats or drop --durable-kv-events)
  instead of an opaque error
- tests: pin DYN_EVENT_PLANE=nats when durable_kv_events=True so the
  JetStream path keeps working under the new ZMQ default
- docs/CLI help updated to reflect ZMQ-default / NATS-opt-in

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@zhongdaor-nv
zhongdaor-nv requested review from a team as code owners June 23, 2026 21:56
@zhongdaor-nv
zhongdaor-nv requested a review from a team June 23, 2026 21:56
@github-actions github-actions Bot added feat documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. labels Jun 23, 2026
@zhongdaor-nv zhongdaor-nv changed the title feat(runtime): default event plane to ZMQ for all backends feat(runtime): default event plane to ZMQ for all backends [WIP] Jun 23, 2026
@github-actions

github-actions Bot commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

@coderabbitai

coderabbitai Bot commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Switches the default DYN_EVENT_PLANE / EventTransportKind from NATS (previously auto-selected based on discovery backend) to ZMQ universally. NATS becomes strictly opt-in via DYN_EVENT_PLANE=nats. The change propagates through the Rust runtime resolution logic, a durable-KV mismatch error message, the Python test fixture's env-var save/restore handling, CLI help strings, and documentation.

Changes

ZMQ as universal default event plane

Layer / File(s) Summary
EventTransportKind default and resolve_event_transport_kind logic
lib/runtime/src/discovery/mod.rs, lib/runtime/src/distributed.rs
#[default] variant on EventTransportKind switched from Nats to Zmq; from_env() and from_env_or_default() return Zmq for unset/empty/invalid DYN_EVENT_PLANE. resolve_event_transport_kind removes is_local() branching and uses DYN_EVENT_PLANE as the sole authoritative source with Zmq as the universal fallback; doc comments updated.
Durable-KV mismatch error and test fixture env-var lifecycle
lib/llm/src/kv_router/indexer/recovery/subscriber.rs, tests/conftest.py
anyhow::bail! message in start_subscriber updated to name the ZMQ default and offer DYN_EVENT_PLANE=nats or removing --durable-kv-events. runtime_services_dynamic_ports fixture now snapshots DYN_EVENT_PLANE before the test, pins it to "nats" when durable_kv_events=True, and restores or removes it at teardown.
CLI help text and documentation
components/src/dynamo/common/configuration/groups/runtime_args.py, components/src/dynamo/frontend/frontend_args.py, docs/design-docs/event-plane.md, docs/components/router/router-guide.md
--event-plane help strings in both Python arg groups drop auto-detection wording and document zmq as the fixed default. Design doc transport table, Transport Selection instructions, and Environment Variables section are rewritten for the ZMQ default / NATS opt-in model; router-guide KV event transport table updated accordingly.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers the overview, detailed changes, and testing validation, but the 'Related Issues' section is incomplete—it neither links to issues nor confirms no related issue exists with the required checkbox. Complete the 'Related Issues' section by either linking to relevant issues (e.g., 'Closes #XXXX') or checking the 'Confirmed — no related issue' checkbox if the PR is not linked to any issue.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: switching the default event plane to ZMQ for all backends.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/conftest.py`:
- Line 1172: Move the function-local import os statement from line 1172 to the
top of the conftest.py file with the other imports. Then, replace all direct
os.environ mutation assignments (lines 1188-1209) with monkeypatch.setenv()
calls. Ensure the fixture receives monkeypatch as a parameter so it can call
monkeypatch.setenv(key, value) instead of os.environ[key] = value for each
environment variable being set.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2a06e340-4619-46ca-bc0a-79e6cbe8a4ee

📥 Commits

Reviewing files that changed from the base of the PR and between f9e8854 and e94f4cf.

📒 Files selected for processing (8)
  • components/src/dynamo/common/configuration/groups/runtime_args.py
  • components/src/dynamo/frontend/frontend_args.py
  • docs/components/router/router-guide.md
  • docs/design-docs/event-plane.md
  • lib/llm/src/kv_router/indexer/recovery/subscriber.rs
  • lib/runtime/src/discovery/mod.rs
  • lib/runtime/src/distributed.rs
  • tests/conftest.py

Comment thread tests/conftest.py Resolved
@datadog-official

This comment has been minimized.

The router e2e tests (mocker/vLLM/SGLang/TRT-LLM) never set DYN_EVENT_PLANE
and relied on the old per-backend default (etcd -> NATS). With ZMQ now the
default, the in-process harness delivers KV events between a separate engine
subprocess and an in-process router over a naive direct ZMQ PUB/SUB path with
no broker, which drops events published before the subscriber connects
(slow-joiner). The router then sees zero KV blocks and the decision/overload/
sync assertions fail.

Pin DYN_EVENT_PLANE=nats in runtime_services_dynamic_ports' etcd branch (where
NATS is already started), reproducing the pre-existing etcd->NATS default for
the whole integration suite. file/mem backends stay on the ZMQ default and
explicit event_plane="zmq" overrides still win.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review feedback on the etcd branch of runtime_services_dynamic_ports:
- Drop the redundant function-local `import os` (os is already imported at
  module top).
- Replace the manual os.environ save/set/restore for NATS_SERVER,
  ETCD_ENDPOINTS, and DYN_EVENT_PLANE with monkeypatch.setenv(), which
  auto-restores at teardown (including any values set by the session-scoped
  fixture) and removes the boilerplate.

monkeypatch is function-scoped like this fixture, so it tears down after the
fixture's server context managers exit; env vars are restored to their prior
(session) values. Behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@zhongdaor-nv
zhongdaor-nv temporarily deployed to external_collaborator June 24, 2026 18:04 — with GitHub Actions Inactive
@zhongdaor-nv zhongdaor-nv changed the title feat(runtime): default event plane to ZMQ for all backends [WIP] feat(runtime): default event plane to ZMQ for all backends Jun 24, 2026

@biswapanda biswapanda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@athreesh athreesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

CI on the ZMQ default flagged 12 failures, all in test_router_e2e_with_mockers.py
(test_router_decisions, test_indexers_sync[nats_core], the overload_503 tests, and
test_router_decisions_disagg_round_robin_prefill_dp_rank). All GPU router e2e tests
(vLLM/SGLang) passed on ZMQ.

Root cause: the mock engine publishes KV events instantly, before the in-process
router's ZMQ subscription connects (ZMQ slow-joiner), so the router sees zero
events. Real engines start slowly enough to avoid this, which is why their router
e2e tests pass on ZMQ and cover the new default.

Add a mocker-scoped autouse fixture that pins DYN_EVENT_PLANE=nats for etcd-backed
mocker tests. file-backed variants keep the ZMQ default; an explicit
DYN_EVENT_PLANE (e.g. durable_kv_events) is left untouched.

Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
black 23.1.0 requires two blank lines between the new autouse fixture and the
module-level pytestmark. No behavior change.

Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the mocker NATS pin with a short ZMQ subscription warm-up so the mocker
tests also exercise the (now default) ZMQ event plane.

Root cause was a ZMQ slow-joiner: the mock engine publishes KV events before the
router's event-plane SUB has connected, so the router missed them and the routing
assertions failed (real engines start slowly enough to avoid this, which is why
the vLLM/SGLang/TRT-LLM router e2e tests already pass on ZMQ).

_zmq_event_plane_warmup() sleeps briefly (only when not on NATS) to let the SUB
attach before the test starts sending. Called from _create_kv_router_with_timeout
(covers the in-process KvRouter tests) and the two overload_503 tests (frontend
HTTP path). The mocker-scoped NATS autouse fixture is removed.

durable_kv_events still pins NATS (JetStream is NATS-only); file-backed variants
keep the ZMQ default.

Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…inning NATS"

This reverts commit 6ef420d. The ZMQ warm-up did not fix the mocker
failures: the real root cause is that a worker running multiple dp_ranks in one
process registers one ZMQ EventChannel per publisher, but they collide on a
per-process discovery key, so the router discovers only one and misses the other
dp_ranks' KV events (a warm-up of any length can't help — verified up to 10s).

Restore the mocker NATS pin so this PR (ZMQ default) stays green, and defer the
multi-publisher-per-process discovery-key fix to a follow-up (candidates: PR
#10972 unique-key, or the share-publishers-across-dp-ranks approach).

Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

This branch was previously deployed

2 inactive deployments
GITLAB — a5c5b3b1 Deployed Jun 26, 2026 by copy-pr-bot[bot] via Trigger CI Pipeline #32158
external_collaborator — a5c5b3b1 Deployed Jun 26, 2026 by zhongdaor-nv via ok-to-test #529
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants