Skip to content

util: add reusable comma-separated env-list parser (NIX-1621) - #1927

Merged
e-eygin merged 3 commits into
ai-dynamo:mainfrom
e-eygin:eeygin/nix-1621-telemetry-reusable-comma-separated-env-list-parser-dedupe
Jul 14, 2026
Merged

e-eygin merged 3 commits into
ai-dynamo:mainfrom
e-eygin:eeygin/nix-1621-telemetry-reusable-comma-separated-env-list-parser-dedupe

Conversation

@e-eygin

@e-eygin e-eygin commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What?

Adds a small, header-only helper nixl::str::splitStripped(absl::string_view, char delim = ',')
in src/utils/common/str_util.h that splits a delimited string into non-empty,
whitespace-stripped tokens (order-preserving, duplicates kept). Refactors
resolveTraceBackends() in src/core/nixl_agent.cpp (NIXL_TRACE_BACKENDS) to use it,
replacing the open-coded absl::StrSplit(',') + absl::StripAsciiWhitespace + skip-empty
loop with a range-insert into the backend set. Adds unit tests
(test/gtest/unit/util/str_util_test.cpp).

Why?

Follow-up from the PR #1897 review (NIX-1621): the same comma-separated env-list parsing
idiom is open-coded in several places (tracing backends, DOCA exporter backends, histogram
buckets, per-metric activation). This lands the shared helper and converts the one call site
that exists on main today (resolveTraceBackends). Kept intentionally minimal and
behavior-preserving so the remaining call sites can adopt it independently as their PRs land;
if this merges first the helper is already available to cherry-pick on rebase, avoiding a
separate follow-up PR.

Ref: https://linear.app/nvidia/issue/NIX-1621

How?

splitStripped centralizes the StrSplit + StripAsciiWhitespace + skip-empty logic and
returns a std::vector<std::string>; deduplication stays a caller concern (resolveTraceBackends
still collects into a std::set, so the set-but-empty "off" semantics and nsys auto-enable
precedence are unchanged). No TELEMETRY_VERSION/API change. Existing tracing tests
(Tracing.ResolveBackends*) plus 4 new StrSplitStripped cases pass.

Summary by CodeRabbit

  • Bug Fixes
    • Improved trace backend configuration parsing to trim ASCII whitespace, ignore empty entries, and correctly handle repeated delimiters while keeping the prior NVTX enable/disable behavior.
  • New Features
    • Added shared string-splitting utilities to split/trim and optionally return a deduplicated set.
  • Tests
    • Added unit tests for the new split behaviors (ordering vs. deduplication, whitespace-only handling, and custom delimiters).
    • Updated the unit-test build setup to include the new utility tests.

Extract the repeated absl::StrSplit(',') + StripAsciiWhitespace +
skip-empty idiom into a header-only nixl::str::splitStripped helper and
use it in resolveTraceBackends (NIXL_TRACE_BACKENDS). Behavior-preserving;
other open-coded call sites can adopt it as their PRs land.

Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
@e-eygin
e-eygin requested a review from a team as a code owner July 11, 2026 12:17
@copy-pr-bot

copy-pr-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@e-eygin e-eygin self-assigned this Jul 11, 2026
@github-actions

Copy link
Copy Markdown

👋 Hi e-eygin! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

@coderabbitai

coderabbitai Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 0e65ad4a-5de1-486e-b27e-922c7aed469b

📥 Commits

Reviewing files that changed from the base of the PR and between 460cb17 and 286bc4d.

📒 Files selected for processing (3)
  • src/core/nixl_agent.cpp
  • src/utils/common/str_util.h
  • test/gtest/unit/util/str_util_test.cpp

📝 Walkthrough

Walkthrough

Adds reusable stripped string-splitting helpers, uses splitStrippedSet for NIXL_TRACE_BACKENDS parsing, and adds Meson wiring plus GoogleTest coverage for filtering, ordering, deduplication, and custom delimiters.

Changes

Trace backend parsing

Layer / File(s) Summary
String utility contract
src/utils/common/str_util.h
Adds splitStripped for ordered tokens and splitStrippedSet for deduplicated tokens, both trimming ASCII whitespace and dropping empty entries.
Trace backend integration
src/core/nixl_agent.cpp
Replaces local Abseil parsing in resolveTraceBackends with splitStrippedSet while preserving explicit-disable and NVTX auto-enable behavior.
Utility test wiring and coverage
test/gtest/unit/meson.build, test/gtest/unit/util/meson.build, test/gtest/unit/util/str_util_test.cpp
Registers utility tests in Meson and covers trimming, empty tokens, ordering, duplicates, sorting, and custom delimiters.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested reviewers: colinnv, ovidiusm, brminich

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately reflects the main change: adding a reusable env-list parser.
Description check ✅ Passed The description follows the template and includes clear What, Why, and How sections with relevant detail.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/utils/common/str_util.h`:
- Around line 17-18: Update the header guard in str_util.h from
NIXL_UTILS_COMMON_STR_UTIL_H to the full path-derived form
NIXL_SRC_UTILS_COMMON_STR_UTIL_H, and use the same corrected macro in both the
`#ifndef` and `#define` directives.
- Around line 29-30: Add a concise Doxygen-style block comment immediately
before the public splitStripped function, documenting its purpose, input string,
delimiter behavior, and returned stripped segments. Keep the existing signature
and implementation unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: ab486496-c6b2-4b01-803e-badd78959c04

📥 Commits

Reviewing files that changed from the base of the PR and between 644facf and 094804d.

📒 Files selected for processing (5)
  • src/core/nixl_agent.cpp
  • src/utils/common/str_util.h
  • test/gtest/unit/meson.build
  • test/gtest/unit/util/meson.build
  • test/gtest/unit/util/str_util_test.cpp

Comment thread src/utils/common/str_util.h Outdated
Comment thread src/utils/common/str_util.h
Match the repo's path-derived header-guard convention
(NIXL_SRC_UTILS_COMMON_STR_UTIL_H) and add a Doxygen block on the public
splitStripped helper. Addresses CodeRabbit review on ai-dynamo#1927.

Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
@e-eygin

e-eygin commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

/build

@e-eygin

e-eygin commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 460cb17

@e-eygin

e-eygin commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@ColinNV @ovidiusm please review

Comment thread src/core/nixl_agent.cpp Outdated
Per review (ColinNV on ai-dynamo#1927): several env-list consumers split *and*
deduplicate, so add a set-returning nixl::str::splitStrippedSet alongside
splitStripped and use it in resolveTraceBackends (drops the manual
set-insert). Deduped call sites can now parse in one call.

Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
@e-eygin
e-eygin requested a review from ColinNV July 14, 2026 08:35
@e-eygin

e-eygin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

/build

@e-eygin

e-eygin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 286bc4d

@svc-nixl

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agentnixl-ci-gpu · commit 11f3dc1a

TL;DR: The "Run Nixlbench tests" stage was killed (exit 143) after ~48 minutes of total silence following the UCCL READ/DRAM→VRAM benchmark — a hang in the UCCL backend where a nixlbench process never exits, leaving parallel --halt now,fail=1 waiting forever. This is a UCCL-backend hang, not a legitimately slow test, so the fix is to add a per-invocation timeout and investigate UCCL teardown/connection handling — not to raise the stage time limit.

Full analysis

Summary: Jenkins nixl-ci-gpu #2716 stage "Run Nixlbench tests" (node 318) aborted with exit code 143 after wall-clock kill; one of the two paired UCCL nixlbench processes hung and never exited.

Root cause: Scanning the log timestamps, the last meaningful application output is [09:31:46.721Z] Engine destroyed at the end of the UCCL READ / initiator=DRAM / target=VRAM iteration. The very next log line is [10:20:01.059Z] Sending interrupt signal to process — a ~48-minute gap of complete silence, i.e. the process hung. The next loop iteration (UCCL READ / DRAM→DRAM, .gitlab/test_nixlbench.sh:96-102) never even printed its "Using address ... for ASIO runtime" banner, meaning the previous run_nixlbench_two_workers_asio invocation's parallel --halt now,fail=1 (line 84) never returned — one of the two UCCL nixlbench worker processes did not terminate after "Engine destroyed", so parallel blocked indefinitely. The UCCL runs also show clear instability just before the hang: throughput collapses to ~0.035 GB/s (vs ~2–5 GB/s for UCX) and repeated Failed to set thread affinity to NUMA node 1 / Error/HUP on connection warnings, consistent with a flaky UCCL connect/teardown path that intermittently deadlocks on shutdown.

Implicated commit: Not the PR under test — PR #1927 does not touch UCCL or the test script. The hang is in the UCCL backend, last substantively changed by [REDACTED:Hex High Entropy String] "Enable local xfer for UCCL backend" (Pravein Govindan Kannan) and [REDACTED:Hex High Entropy String] "UCCL: Simplify and Optimize for batch transfers." The test harness lacks any per-run timeout (.gitlab/test_nixlbench.sh, no timeout wrapper), which is what turns the hang into a wall-clock kill.

File: .gitlab/test_nixlbench.sh:80-102 (UCCL loop + parallel invocation with no timeout); underlying defect in the UCCL plugin shutdown path (src/plugins/uccl).

Suggested fix:

  1. Immediate CI robustness: wrap the paired benchmark in a timeout so a UCCL hang fails fast instead of consuming the whole stage, e.g. in run_nixlbench_two_workers_asio change line 84 to parallel --line-buffer --timeout 300 --halt now,fail=1 ::: ... (or wrap each nixlbench invocation in timeout 300). This converts the silent 48-min hang into a diagnosable failure.
  2. Root-cause: investigate the UCCL engine teardown / next-connection path — after "Engine destroyed" one worker process fails to exit. Check whether a UCCL background accept/event-loop thread (see Waiting to accept incoming connection... / Error/HUP on connection in rdma/epoll_client.h) is not joined on shutdown, causing the process to block on exit. The Failed to set thread affinity to NUMA node 1 errors also suggest the UCCL engine mis-initializes on this node and should be handled gracefully.

Related: UCCL backend PRs #1428, #1271, #1151; consistency-check PR #1103. No existing open issue matches this specific hang.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 61cfdcc2-fd7f-4614-8f21-2bca35bd556b in the triage console for the audit trail.

@e-eygin
e-eygin enabled auto-merge (squash) July 14, 2026 10:49
@e-eygin
e-eygin merged commit e2b2576 into ai-dynamo:main Jul 14, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants