Skip to content

feat(observability): Export forward pass metrics to local collectors - #11274

Closed
nealvaidya wants to merge 2 commits into
mainfrom
codex/fpm-tachometer-pr
Closed

feat(observability): Export forward pass metrics to local collectors#11274
nealvaidya wants to merge 2 commits into
mainfrom
codex/fpm-tachometer-pr

Conversation

@nealvaidya

Copy link
Copy Markdown
Contributor

Summary

  • expose event-plane envelope metadata from FpmEventSubscriber through a new recv_envelope() API while preserving the existing recv() behavior
  • add dynamo.common.export_forward_pass_metrics, an opt-in subscriber that discovers one or more Dynamo components and forwards decoded forward-pass metrics as NDJSON over a local Unix socket
  • preserve publisher ID, event sequence, publication timestamp, component, namespace, worker ID, and DP rank for downstream integrity checks and time alignment
  • add a focused serialization test for the local export record

Motivation

Benchmark-scoped collectors need the full forward-pass event history from Dynamo rather than a backend-specific vLLM connection or tracking mode's latest snapshot. Keeping discovery and decoding in Dynamo provides one backend-neutral integration point for vLLM, SGLang, and TensorRT-LLM publishers.

The local socket boundary also lets a collector run without NATS: it can use etcd discovery, the configured Dynamo event plane (for example ZMQ), and a TCP request-plane runtime solely to initialize the distributed runtime.

Compatibility

  • FpmEventSubscriber.recv() remains unchanged for existing callers.
  • recv() and recv_envelope() share the existing receive mode and remain mutually exclusive with start_tracking().
  • The exporter is opt-in and does not change worker or frontend startup behavior.

Validation

  • cargo fmt --all -- --check
  • cargo check --manifest-path lib/bindings/python/Cargo.toml --lib
  • Ruff format and lint checks for the exporter and test
  • focused pytest for test_export_forward_pass_metrics.py

Signed-off-by: Neal Vaidya <nealv@nvidia.com>
@datadog-official

datadog-official Bot commented Jul 6, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 2 Pipeline jobs failed

PR | backend-status-check   View in Datadog   GitHub Actions

PR | dynamo-status-check   View in Datadog   GitHub Actions

ℹ️ Info

🎯 Code Coverage (details)
Patch Coverage: 31.58%
Overall Coverage: 40.38% (-6.67%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d9a419f | Docs | Give us feedback!

@nealvaidya nealvaidya changed the title [codex] Export forward pass metrics to local collectors feat(observability): Export forward pass metrics to local collectors Jul 6, 2026
@github-actions github-actions Bot added the feat label Jul 6, 2026
Signed-off-by: Neal Vaidya <nealv@nvidia.com>
@nealvaidya

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #11110, which persists forward-pass metrics at the producer as rotating JSONL gzip trace segments. The downstream SRT-Slurm and Tachometer integrations now consume that official trace format, so Dynamo no longer needs the in-process subscriber/exporter added by this draft.

@nealvaidya nealvaidya closed this Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant