Repository navigation
telemetry: native multi-process Prometheus aggregation (prometheus_mp) #1920
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
e-eygin
merged 76 commits into
ai-dynamo:main
from
e-eygin:eeygin/nix-1614-native-mp-prometheus-aggregation
Aug 6, 2026
Merged
Changes from 10 commits
Commits
Show all changes
76 commits
Select commit
Hold shift + click to select a range
044999a
telemetry: add multiprocess metric-state store (mp_store)
e-eygin f17b312
telemetry: add multiprocess scrape-time collector (mp_collector)
e-eygin 5f71e4e
telemetry: label mp series by pid to guarantee uniqueness
e-eygin 000eaa3
telemetry: add prometheus_mp exporter, plugin, and bind-race election
e-eygin 30f12dd
telemetry: add prometheus_mp multiprocess e2e forking test
e-eygin 38b5093
telemetry: harden mp collector against mid-scrape churn and orphans
e-eygin 2a624a5
docs: document the prometheus_mp multi-process telemetry exporter
e-eygin 0fa45a1
docs: add prometheus_mp plugin README
e-eygin 93f95ae
docs: align prometheus_mp shared-dir guidance with Dynamo
e-eygin b4093af
telemetry: fix codespell (reuse nixlTime::getNs, unparsable spelling)
e-eygin 3bf2139
telemetry: address CodeRabbit review (mp bounds guard, dir iteration,…
e-eygin cd22887
telemetry: rename mp rank label dp_rank -> local_rank
e-eygin 4e69307
telemetry: remove mp store file on clean writer destruction
e-eygin f5de93f
telemetry: add agent_instance label so same-name mp agents stay distinct
e-eygin 4c74834
docs: clarify mp stale TTL is from last update and document agent_ins…
e-eygin ad437a8
docs: note the status label on agent_errors_total in telemetry.md
e-eygin aeed305
Merge branch 'main' into eeygin/nix-1614-native-mp-prometheus-aggrega…
e-eygin aa3f6bd
Merge branch 'main' into eeygin/nix-1614-native-mp-prometheus-aggrega…
e-eygin 4f7036f
telemetry: address prometheus_mp review (ordering, RAII, const, owner_)
e-eygin e982ed3
telemetry: centralize getHostname() into common/hostname.h
e-eygin 6452ac5
telemetry: drop redundant mp prefix from prometheus_mp store types
e-eygin 1cc0fdc
telemetry: check ::write result in prometheus_mp e2e test
e-eygin 0d3ecac
telemetry: prometheus_mp review cleanups (dead API, TTL default, stor…
e-eygin 87b1b94
docs: prometheus_mp private-dir and shared-PID-namespace requirements
e-eygin 8162099
telemetry: prometheus_mp test hygiene + plugin-loader coverage
e-eygin b7c6380
telemetry: don't reap a live prometheus_mp peer on a transient read e…
e-eygin 557c0e6
telemetry: harden prometheus_mp store reads against exceptions
e-eygin b90bc43
telemetry: histogram support for the prometheus_mp exporter
e-eygin 6d0f506
telemetry: fix prometheus_mp plugin-loader test, document owner death
e-eygin fb0a329
telemetry: cut redundant clock reads and zero loads in prometheus_mp
e-eygin 9f18292
telemetry: harden the prometheus_mp e2e test and fix doc/header nits
e-eygin 748ffcd
telemetry: gate prometheus_mp histogram writes on the descriptor
e-eygin 33b1f4e
test: register the prometheus plugin dir once per process
e-eygin efa2818
telemetry: assert stale-store reaping, plus doc and naming fixes
e-eygin 25216b6
telemetry: scope the exporter parity claim to names and semantics
e-eygin 9151633
telemetry: say metric definitions, not series, in the parity claim
e-eygin 728c5a5
telemetry: ignore prometheus_mp stores owned by another user
e-eygin 94baeff
telemetry: centralize the agent_errors_total family name and help
e-eygin dbea2d9
telemetry: tell a sibling owner from a stranger on the scrape port
e-eygin b7dbc34
telemetry: elect the prometheus_mp owner by lock, not by the bind
e-eygin 1fb9fbf
telemetry: harden the prometheus_mp owner election
e-eygin 536ebee
telemetry: warn when the prometheus_mp lock file cannot be opened
e-eygin 8240177
telemetry: ignore a foreign-owned prometheus_mp lock file
e-eygin 40d3ea7
telemetry: clear the prometheus_mp lock file when the election is won
e-eygin 78a6dac
telemetry: say the election locks a file, and skip when chown is denied
e-eygin 2baf0ac
telemetry: qualify the prometheus_mp single-binder guarantee
e-eygin 2c9a97c
telemetry: correct the prometheus_mp stale-handling description
e-eygin a46cf4c
telemetry: bound stale retention by the scrape, not the TTL alone
e-eygin 0636c7d
telemetry: correct exporter counts and gaps in the docs
e-eygin 540cd15
prometheus_mp: harden the store, the clock and the telemetry directory
e-eygin 6c814cc
prometheus_mp: take the endpoint over when the owner dies
e-eygin f01922b
prometheus_mp: mark refreshHeartbeat() nodiscard
e-eygin c629aa1
prometheus_mp: give the on-disk store format its own header
e-eygin 01b4831
prometheus_mp: move endpoint ownership into its own class
e-eygin 33251de
prometheus_mp: give the election back when a takeover fails
e-eygin b715102
Merge remote-tracking branch 'ai-dynamo/main' into eeygin/nix-1614-na…
e-eygin 1b7d8f4
prometheus_mp: keep the election when reclaim() runs on the owner
e-eygin 80c5a05
prometheus_mp: refuse to move over an election that holds the lock
e-eygin 7c1dfcd
prometheus_mp: guard the election move on the lock, not the descriptor
e-eygin 0f615dd
prometheus_mp: give the test telemetry dirs the 0700 they ask operato…
e-eygin bf0ff19
prometheus_mp: document the election move operations
e-eygin 121573d
prometheus_mp: stop the telemetry tests depending on timing and lookups
e-eygin 799e4a4
prometheus_mp: say what a refused election move leaves behind
e-eygin 4570b6d
Merge remote-tracking branch 'ai-dynamo/main' into eeygin/nix-1614-na…
e-eygin f3deeb9
prometheus_mp: drop TIME_WAIT from the takeover comment
e-eygin 72a07b9
prometheus_mp: make an election impossible to overwrite in place
e-eygin fb1aecb
prometheus_mp: publish the owner endpoint as one fixed-size record
e-eygin 58a0642
utils: move scopedFd out of the prometheus_mp plugin
e-eygin fa6ab30
Merge remote-tracking branch 'ai-dynamo/main' into eeygin/nix-1614-na…
e-eygin 6a39475
telemetry: keep the prometheus_mp store on clean shutdown
e-eygin 22c64d4
telemetry: elect the prometheus_mp owner per endpoint
e-eygin f2f6586
telemetry: name prometheus_mp bind strings addresses, not endpoints
e-eygin 5f92735
telemetry: use C++20 starts_with/ends_with for prometheus_mp name checks
e-eygin b9ccedb
telemetry: decide prometheus_mp store liveness by lock, not by pid
e-eygin 572458f
telemetry: state what an unlockable prometheus_mp store really costs
e-eygin 8f25980
prometheus_mp: RAII the store mapping and drop the /proc start-time read
e-eygin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| <!-- | ||
| SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| --> | ||
|
|
||
| # NIXL multi-process Prometheus Telemetry exporter plug-in (`prometheus_mp`) | ||
|
|
||
| `prometheus_mp` exposes the telemetry of **all** processes of a multi-process | ||
| NIXL run (e.g. tensor/data parallelism) behind a **single** Prometheus scrape | ||
| endpoint, without any DOCA/DTS dependency. | ||
|
|
||
| It complements the single-process [`prometheus`](../prometheus/README.md) exporter | ||
| (which binds one port per process, so only one rank's metrics are scraped) and the | ||
| DOCA/CollectX exporter (which aggregates via an external DTS service). Use | ||
| `prometheus_mp` when you want all ranks aggregated natively with no extra | ||
| infrastructure. General NIXL telemetry background: [docs/telemetry.md](../../../../docs/telemetry.md). | ||
|
|
||
| ## Dependencies | ||
|
|
||
| Same as the `prometheus` plug-in: the bundled prometheus-cpp subproject and | ||
| `libcurl` (`libcurl4-openssl-dev` / `libcurl-devel`). | ||
|
|
||
| ## How it works | ||
|
|
||
| - **Every process writes its own metric state** to a per-process memory-mapped | ||
| file in a shared directory (`NIXL_TELEMETRY_MULTIPROC_DIR`). Updates are | ||
| lock-free; there is no serialization. | ||
| - **Bind-race owner election.** On startup each process tries to bind the scrape | ||
| port. The one that wins ("owner") runs the HTTP endpoint plus a collector that, | ||
| on each scrape, reads every live process's file and republishes them as one | ||
| exposition. The processes that lose the race run in **writer-only** mode (no | ||
| HTTP server). A bind collision is therefore benign -- every process gets a valid | ||
| telemetry sink; no rank is dropped and no scary error is logged. | ||
| - **Per-process series.** Each process is exported as its own series (cumulative | ||
| counters and last-operation gauges), never summed across processes, so | ||
| per-process values stay correct and monotonic. | ||
| - **Stale handling.** When a process exits, the owner drops its series once the | ||
| process is gone (verified by pid + `/proc` start time) or its data ages past the | ||
| TTL, and reaps the file. Cleanup is performed by the owner, since a killed | ||
| process cannot clean up after itself. | ||
|
|
||
| ## Configuration | ||
|
|
||
| ```bash | ||
| export NIXL_TELEMETRY_ENABLE="y" | ||
| export NIXL_TELEMETRY_EXPORTER="prometheus_mp" # selects libtelemetry_exporter_prometheus_mp.so | ||
| export NIXL_TELEMETRY_MULTIPROC_DIR="/tmp/nixl_metrics" # REQUIRED: shared by all ranks in the pod | ||
| ``` | ||
|
|
||
| This mirrors Dynamo's `PROMETHEUS_MULTIPROC_DIR` convention (a shared folder that | ||
| every related process writes into, one leader exports): all ranks that should be | ||
| aggregated together must point `NIXL_TELEMETRY_MULTIPROC_DIR` at the **same** | ||
| directory. Unlike Dynamo -- which auto-creates a temp dir in the parent and lets | ||
| child engine processes inherit it -- NIXL is a library loaded independently in each | ||
| rank, so there is no parent to propagate the path; the launcher/operator must set | ||
| the same directory for every rank (hence it is required, not auto-defaulted, so a | ||
| per-process temp dir can never silently break aggregation). | ||
|
|
||
| Recommended, following Dynamo's model: a shared **local** folder, one per pod / | ||
| process-family, treated as ephemeral (e.g. a per-pod Kubernetes `emptyDir`, or a | ||
| temp dir cleaned between runs). It must be a local filesystem -- **not** a network | ||
| filesystem (NFS/CIFS), where mmap `MAP_SHARED` cross-process visibility is not | ||
| guaranteed (the same restriction Dynamo's multiprocess dir has). tmpfs (e.g. a | ||
| Memory-medium `emptyDir` or `/dev/shm`) works and avoids any disk writeback, but is | ||
| optional -- a plain local dir is fine, since updates hit the page cache and the | ||
| per-process store files are ~one page each. | ||
|
|
||
| ### Optional configuration | ||
|
|
||
| ```bash | ||
| # Scrape port (default 9090) and bind scope -- shared with the prometheus plug-in. | ||
| export NIXL_TELEMETRY_PROMETHEUS_PORT="<port_num>" | ||
| export NIXL_TELEMETRY_PROMETHEUS_LOCAL="y" # bind 127.0.0.1 instead of 0.0.0.0 | ||
|
|
||
| # Optional dp_rank label: names the env var that holds the rank (default LOCAL_RANK). | ||
| # If that env var is unset, no dp_rank label is emitted (series stay unique via pid). | ||
| export NIXL_TELEMETRY_RANK_ENV="LOCAL_RANK" | ||
|
|
||
| # Seconds after which a dead process's store is considered stale and reaped (default 30). | ||
| export NIXL_TELEMETRY_MP_STALE_TTL="30" | ||
| ``` | ||
|
|
||
| ## Metric labels | ||
|
|
||
| Every series is labeled by: | ||
|
|
||
| - `hostname` -- host where the agent runs. | ||
| - `agent_name` -- the agent name given at initialization. | ||
| - `pid` -- the producing process id. This guarantees each process is a distinct | ||
| series even if agent names collide; it is deliberately **not** named `instance` | ||
| (a reserved Prometheus target label). | ||
| - `dp_rank` -- **optional**, present only when a rank env var (see | ||
| `NIXL_TELEMETRY_RANK_ENV`) is set. | ||
| - `status` -- only on `agent_errors_total`, bounded by the fixed `AGENT_ERR_*` set. | ||
|
|
||
| The metric names, types, counter/gauge semantics, and events are identical to the | ||
| single-process [`prometheus`](../prometheus/README.md) exporter (same shared | ||
| descriptor). | ||
|
|
||
| ## Design scope & limitations | ||
|
|
||
| This exporter is **purpose-built for NIXL's telemetry model, not a generic | ||
| Prometheus multiprocess store** (in particular it is not compatible with, and does | ||
| not reuse, Python `prometheus_client`'s multiprocess format): | ||
|
|
||
| - The metric set is fixed at compile time; slots are positional, so metric names | ||
| are never stored in the files. | ||
| - Per-process label values (`hostname`, `agent_name`, `pid`, `dp_rank`) are | ||
| captured once at startup and never change. Events carry only a numeric value -- | ||
| there are **no per-observation labels**. | ||
| - Consequently the store **cannot represent a metric with a dynamic / | ||
| high-cardinality label** whose value varies per observation. No NIXL metric has | ||
| such a label today; if one is ever added, this exporter would need a different | ||
| (keyed) store. | ||
|
|
||
| This is the native, dependency-free path. For aggregation through an external | ||
| telemetry service, use the DOCA/CollectX exporter (IPC to DTS) instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # Multiprocess metric-state store: a small, prometheus-cpp-free component (mmap | ||
| # per-process store) shared by the prometheus_mp plugin and its unit tests. Built | ||
| # unconditionally so the CTest coverage runs even where prometheus-cpp is absent; | ||
| # the plugin shared library (which needs prometheus-cpp) is added in a later step. | ||
| prometheus_mp_inc_dirs = include_directories('.') | ||
|
|
||
| telemetry_mp_store_lib = static_library( | ||
| 'nixl_telemetry_mp_store', | ||
| 'mp_store.cpp', | ||
| include_directories: [nixl_inc_dirs, utils_inc_dirs, prometheus_mp_inc_dirs], | ||
| dependencies: [nixl_common_dep], | ||
| install: false, | ||
| pic: true, | ||
| ) | ||
|
|
||
| nixl_telemetry_mp_store_dep = declare_dependency( | ||
| link_with: telemetry_mp_store_lib, | ||
| include_directories: prometheus_mp_inc_dirs, | ||
| dependencies: [nixl_common_dep], | ||
| ) | ||
|
|
||
| # Scrape-time collector: converts peer store snapshots into prometheus-cpp metric | ||
| # families. Needs prometheus-cpp (core), so it is gated on it being available | ||
| # (same condition as the prometheus plugin, set in the sibling subdir above). | ||
| if prometheus_found | ||
| telemetry_mp_collector_lib = static_library( | ||
| 'nixl_telemetry_mp_collector', | ||
| 'mp_collector.cpp', | ||
| include_directories: [nixl_inc_dirs, utils_inc_dirs, prometheus_mp_inc_dirs], | ||
| dependencies: [nixl_common_dep, prometheus_core_dep, nixl_telemetry_mp_store_dep], | ||
| install: false, | ||
| pic: true, | ||
| ) | ||
|
|
||
| nixl_telemetry_mp_collector_dep = declare_dependency( | ||
| link_with: telemetry_mp_collector_lib, | ||
| include_directories: prometheus_mp_inc_dirs, | ||
| dependencies: [nixl_common_dep, prometheus_core_dep, nixl_telemetry_mp_store_dep], | ||
| ) | ||
|
|
||
| # Exporter with bind-race election. Built as a static library too so it can be | ||
| # unit-tested directly (in addition to being packaged into the plugin .so). | ||
| telemetry_mp_exporter_lib = static_library( | ||
| 'nixl_telemetry_mp_exporter', | ||
| 'prometheus_mp_exporter.cpp', | ||
| include_directories: [nixl_inc_dirs, utils_inc_dirs, prometheus_mp_inc_dirs], | ||
| dependencies: [nixl_common_dep, prometheus_core_dep, prometheus_pull_dep, | ||
| nixl_telemetry_mp_store_dep, nixl_telemetry_mp_collector_dep], | ||
| install: false, | ||
| pic: true, | ||
| ) | ||
|
|
||
| nixl_telemetry_mp_exporter_dep = declare_dependency( | ||
| link_with: telemetry_mp_exporter_lib, | ||
| include_directories: prometheus_mp_inc_dirs, | ||
| dependencies: [nixl_common_dep, prometheus_core_dep, prometheus_pull_dep, | ||
| nixl_telemetry_mp_store_dep, nixl_telemetry_mp_collector_dep], | ||
| ) | ||
|
|
||
| # Installable plugin: selected via NIXL_TELEMETRY_EXPORTER=prometheus_mp. | ||
| prometheus_mp_exporter_plugin = shared_library( | ||
| 'libtelemetry_exporter_prometheus_mp', | ||
| 'prometheus_mp_plugin.cpp', | ||
| include_directories: [nixl_inc_dirs, utils_inc_dirs, prometheus_mp_inc_dirs], | ||
| dependencies: [nixl_infra, nixl_common_dep, absl_log_dep, prometheus_core_dep, | ||
| prometheus_pull_dep, nixl_telemetry_mp_exporter_dep], | ||
| install: true, | ||
| install_dir: plugin_install_dir, | ||
| name_prefix: '', | ||
| ) | ||
| message('Building Prometheus multiprocess telemetry exporter plugin') | ||
| endif |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.