docs: refresh tracing.md after NVTX completeness + correlation (#1852)

[e-eygin]

What?

Update docs/tracing.md to match the tracing code after #1852 (NVTX completeness

• cross-thread correlation) landed in main. Docs-only, no code changes.

Why?

Several sections predated #1852 and were stale/misleading:

• The instrumented-operations table was missing the metadata-exchange and
  mem-view call sites.
• It still described NVTX attributes as lossy key=value marks.
• It listed cross-thread correlation as a planned no-op.
• "Planned work" still listed the now-completed NVTX work.

How?

• Add loadRemoteMD, fetchRemoteMD, prepMemView, releaseMemView to the
  instrumented-operations table (with their Kind + attributes).
• NVTX now attaches attributes as typed payloads (nvtxRangePopPayload),
  not key=value marks.
• Cross-thread correlation is implemented: pushCorrelationId()/
  popCorrelationId() + a request-handle-keyed correlation scope, recorded as
  the NVTX uint64 event payload, so a postXferReq and its xfer.complete
  share an id regardless of the emitting thread.
• Planned-work cleanup: drop the done "NVTX completeness"; note UCX
  backend-engine sub-spans are not planned; keep Chakra + cross-rank correlation.
• Minor wording fix ("here" → "In NIXL tracing").

Test plan

Docs-only; no build/test impact. Rendered/reviewed the Markdown; the
instrumented-ops table and attributes were cross-checked against the call sites
in src/core/nixl_agent.cpp`.```

Summary by CodeRabbit

• Documentation
  • Updated tracing docs to reflect additional covered operations and current correlation behavior.
  • Clarified how tracing backends display attributes, including NVTX payload formatting.
  • Expanded the correlation section with clearer guidance on cross-thread correlation handling.
  • Revised the planned work section to better reflect current tracing roadmap and remove outdated items.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi e-eygin! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 8695a5a9-6a5b-4404-9c5e-c8656ecb24ff

📥 Commits

Reviewing files that changed from the base of the PR and between 471a64e and 2d05d97.

📒 Files selected for processing (1)

• docs/tracing.md

---

📝 Walkthrough

Walkthrough

Updates docs/tracing.md to document expanded instrumented operations, NVTX typed payload attributes replacing key=value marks, expanded cross-thread correlation behavior using pushCorrelationId()/popCorrelationId(), and revised planned work items.

Changes

Tracing documentation

Layer / File(s)	Summary
Instrumented operations and NVTX attribute rendering docs/tracing.md	Adds remote metadata and memory-view operation entries to the instrumented operations list, and updates the description of NVTX attribute rendering to typed payload attributes instead of key=value marks, while dependencies remain ignored.
Correlation semantics and planned work docs/tracing.md	Clarifies the two meanings of "correlation" in NIXL, details cross-thread correlation via correlation scopes keyed on transfer-request handle address with uint64 NVTX payload recording, and revises planned work items around Chakra backend serialization and cross-rank correlation.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

• ai-dynamo/nixl#1852: Implements the NVTX typed payload attributes and cross-thread correlation scope/id behavior that this PR's documentation describes.

Suggested labels: size/L

Suggested reviewers: ovidiusm, ColinNV

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the docs refresh around tracing, NVTX completeness, and correlation.
Description check	✅ Passed	The description follows the template well and covers What, Why, How, and a test plan for this docs-only change.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[e-eygin]

/build

[e-eygin]

/ok to test 2d05d97

[svc-nixl]

👀 Investigating nixl-ci-test-sanitizers build 261…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit d54a224d

I now have a confident diagnosis. The evidence is conclusive:

1. What failed: ucx_tracing_no_pt/TestTransferTracing.NvtxNotifications/0 — the only failed test (exit 42), inside the nixl:sanitizer / gtest suite.
2. Why it failed: The LogProblemCounter absl log-sink (common.cpp:180-205) fails the test whenever a warning/error-severity log message is emitted that isn't covered by a LogIgnoreGuard. During NvtxNotifications, 4 UCX AM send failed with status -16 (Request canceled) errors were logged from ucx_utils.cpp:204 (visible at 07:31:36.94 in stderr, immediately followed by Problem count is 4). These are benign UCX teardown-race errors (in-flight active-message send cancelled during agent destruction), not real failures — the transfers themselves all reported Data verification: OK.
3. The NvtxNotifications test was just added/modified by commit 471a64e (PR tracing: NVTX completeness + cross-thread correlation #1852, same-day as the build) which introduced NVTX cross-thread correlation and this new test case, without a LogIgnoreGuard for the "Request canceled" teardown error.

Timestamps show continuous activity — this is a genuine assertion failure, not a hang or wall-clock kill.

Summary: Sanitizer gtest suite failed because TestTransferTracing.NvtxNotifications/0 tripped the "unexpected NIXL warning/error" guard (exit 42, problem count 4).

Root cause: During the new NvtxNotifications test, agent teardown cancelled in-flight UCX active-message sends, producing 4 UCX AM send failed with status -16 (Request canceled) errors logged at ucx_utils.cpp:204. These benign teardown-race errors are counted by gtest::LogProblemCounter (test/gtest/common.cpp:180) because the test has no LogIgnoreGuard covering that message, so the test fails even though all data transfers verified OK. The test was newly introduced without an ignore rule for expected shutdown noise.

Implicated commit: 471a64e — "tracing: NVTX completeness + cross-thread correlation (#1852)", author e-eygin (dated 2026-07-03, same day as build #261).

File: test/gtest/test_transfer.cpp (the NvtxNotifications test case); guard mechanism in test/gtest/common.cpp:180-205; error origin ucx_utils.cpp:204.

Suggested fix: In the NvtxNotifications test body (and any tracing test that tears down agents with transfers potentially still in flight), add a LogIgnoreGuard for the expected teardown message, e.g. gtest::LogIgnoreGuard ignore("UCX AM send failed with status -16 \\(Request canceled\\)"); scoped over the teardown/agent-destruction phase. Alternatively, ensure the test drains/completes or aborts all outstanding transfers (and syncs the progress path) before destroying the agents so the cancel path isn't hit. Do not raise the timeout — the run was making progress; this is a missing log-ignore rule for a benign race.

Related: PR #1852 (root cause); PR #1288 (introduced the "fail on unexpected error/warning log" mechanism); PR #1709 (added the sanitizer CI builds).

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2508…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit d54a224d

I now have a clear picture. Let me confirm the key detail: the segfaulting case is --backend UCCL --op_type READ --initiator_seg_type VRAM --target_seg_type DRAM. The prior case (VRAM→VRAM) passed. Note the crash is in a two-worker separate-process run, so is_same_process is false, and the target is DRAM (has_ipc = false).

Looking at postXfer, when use_ipc is false (the DRAM cross-process case), ipc_ptrs stays empty but is still passed to uccl_engine_read_vector(...). Meanwhile in prepXfer, ipc_infos is only resized when is_local_conn is true, and use_ipc is only set for the has_ipc or is_same_process cases. The cross-process local DRAM path (is_local_conn && !has_ipc && !is_same_process) is explicitly documented as "falls back to RDMA" — but the code passes an empty ipc_ptrs vector into the read vector call, which the UCCL engine likely dereferences.

Summary: The nixl-ci-gpu build #2508 failed because the nixlbench two-worker test crashed with SIGSEGV (exit 139) on the UCCL / READ / initiator=VRAM / target=DRAM case; a separate Python-test timeout on create_backend("UCX") also occurred.

Root cause: In the UCCL backend, cross-process local transfers to DRAM targets are the one path that is neither has_ipc (GPU/IPC) nor is_same_process. For that path prepXfer never sets use_ipc and never populates ipc_infos, so it is meant to "fall back to RDMA." However postXfer unconditionally passes the ipc_ptrs vector (which is empty in this fallback) into uccl_engine_read_vector(...). The VRAM→VRAM case immediately before it passed (it has IPC), and the VRAM→DRAM case segfaults — consistent with the engine dereferencing per-descriptor IPC pointers that were never provided for the RDMA-fallback DRAM path. The crash surfaces in postXfer/uccl_engine_read_vector (src/plugins/uccl/uccl_backend.cpp:655-658).

Implicated commit: [REDACTED:Hex High Entropy String] — "Enable local xfer for UCCL backend (#1428)", Pravein Govindan Kannan (introduced the IPC/local-xfer + has_ipc/is_same_process branching that leaves the cross-process DRAM path under-specified). The ipc_ptrs passing logic is in postXfer (lines 645-663).

File: src/plugins/uccl/uccl_backend.cpp:556-566 (prepXfer IPC-info population) and src/plugins/uccl/uccl_backend.cpp:645-663 (postXfer passing ipc_ptrs).

Suggested fix: Make the cross-process local DRAM case explicitly take the RDMA path instead of a partially-initialized IPC path. Concretely: in postXfer, only pass a non-empty ipc_ptrs when use_ipc is set and ipc_infos has exactly lcnt populated entries; otherwise pass an empty/sentinel that the UCCL engine treats as pure RDMA. Also add a guard in prepXfer so that for is_local_conn && !rmd->has_ipc && !is_same_process the request is flagged as RDMA-only (do not resize/leave ipc_infos partially filled), and verify uccl_engine_read_vector/write_vector tolerate an empty ipc_ptrs. Add a two-worker UCCL VRAM→DRAM READ/WRITE regression test to the C++ suite.

(Note: the Python-side test_empty_notif_tag 5s timeout on create_backend("UCX") is a separate, likely-flaky failure and should be triaged independently; the SIGSEGV is the primary build-breaker.)

Related: PR #1428 (introduced local UCCL xfer); PR #1595 (unified get-param touched this plugin). No existing issue found specific to this segfault.

Note: the build log contained what looks like an Azure storage account key on the test_azure.sh lines (an AccountKey=... value and AZ_ACCOUNT_KEY). Even though it appears to be a throwaway Azurite dev credential, treat it as exposed and rotate/scrub it from CI logs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 1753fb9a-4db8-44a7-8c6e-281390ce5cff in the triage console for the audit trail.

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu-ep build 77…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit d54a224d

I have the log. The container image built successfully for both ucx-v1.21.x and ucx-master variants. The failure occurred in the "Allocate DL EP Environment" stage. Let me identify the exact failure signature.

Looking at the log:

• The Docker/build phase completed fully (both images pushed to artifactory)
• The pipeline then enters "Allocate DL EP Environment" stage which runs a Slurm salloc over SSH

The failure signature:

salloc: Pending job allocation 1456347
salloc: job 1456347 queued and waiting for resources
salloc: error: Unable to allocate resources: Connection timed out

Both branches (ucx-v1.21.x and ucx-master) failed identically at salloc. Let me examine the timestamps to confirm this is a resource-availability/timeout issue rather than an application hang.

The salloc was invoked at 07:46:18 (v1.21.x) and 07:49:29 (master), with --immediate=3600 (wait up to 3600s = 1 hour for immediate allocation). The failures printed at 08:46:26 and 08:49:38 respectively — almost exactly one hour later. This is the --immediate=3600 timeout expiring: the job sat queued and waiting for resources for the full hour, then Slurm returned "Unable to allocate resources: Connection timed out."

This is a cluster resource-availability issue, not a code/build defect. The job stayed queued the entire hour because no gb200nvl72_cx8 nodes were free on dlcluster.nvidia.com under the blackwell account. This is infrastructure, and it is unrelated to PR #1882's code — the build itself passed completely.

Summary: The nixl-ci-dl-gpu-ep build failed in the "Allocate DL EP Environment" stage — the Slurm salloc on the GB200 cluster could not obtain a node within the 1-hour --immediate window.

Root cause: Cluster resource starvation, not a code defect. Both parallel branches (ucx-v1.21.x and ucx-master) submitted salloc -N 1 -p gb200nvl72_cx8 --immediate=3600 --time=01:30:00 --account=blackwell to dlcluster.nvidia.com. The jobs (1456347, 1456399) were accepted and sat queued and waiting for resources for the full 3600 s, then Slurm returned error: Unable to allocate resources: Connection timed out. The container image build phase preceding this completed successfully and both images were pushed to artifactory, so this is not a build/compile failure. The ~1-hour gap here is the deliberate --immediate=3600 wait for a free node — the process was legitimately blocked on external resource availability (no free gb200nvl72_cx8 nodes), not hung on an application operation.

Implicated commit: unknown — not code-related. The build of commit d54a224 succeeded; the failure is in scheduler/infrastructure.

File: unknown (Jenkins pipeline Slurm salloc step in the "Allocate DL EP Environment" stage; not a repo source file).

Suggested fix:

• Re-run the job; GB200 gb200nvl72_cx8 capacity under the blackwell account was unavailable for the full hour.
• If this recurs, check partition/account queue depth (squeue -p gb200nvl72_cx8, sinfo -p gb200nvl72_cx8) and node health on dlcluster.nvidia.com.
• Consider making the allocation failure clearly distinguishable from a test failure (e.g., mark it as an infra/retryable stage) and/or raising --immediate or adding automatic requeue so transient capacity shortages don't fail the PR.
• Note this failure is independent of PR docs: refresh tracing.md after NVTX completeness + correlation (#1852) #1882's changes — the compile/link/install all passed.

Related: none found.

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu-ep build 79…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit d54a224d

I have the logs. Let me analyze the failure. The master UCX branch build/test track completed all EP elastic tests successfully at 12:08:07. The failure is in a different parallel branch: build_helper_dl_ep/aarch64/ucx-v1.21.x/2.

Let me trace the relevant lines.

The v1.21.x branch attempted a SLURM allocation:

• At 11:59:48 it ran salloc ... --job-name=nixl-ci-ep-v1.21.x-79 --immediate=3600 --time=01:30:00
• At 12:59:58 (exactly ~60 minutes later) it printed:
  • salloc: job 1458051 queued and waiting for resources
  • salloc: error: Unable to allocate resources: Connection timed out
  • Failed in branch build_helper_dl_ep/aarch64/ucx-v1.21.x/2

This is a SLURM scheduling/resource issue, not a code defect. Let me confirm the timing and that no application/code output is implicated.

The evidence is conclusive from the log itself — no source investigation is needed because this is an infrastructure resource-allocation failure, not a code or test failure.

Timeline analysis:

• 11:59:48 — v1.21.x branch issues salloc on partition gb200nvl72_cx8 with --immediate=3600 (wait up to 3600s / 60min for resources).
• The master branch got its allocation immediately (job 1458049, node gb-nvl-118-compute07) and ran all EP elastic tests to completion successfully by 12:08:07.
• 12:59:58 (~60 min later, matching --immediate=3600) — the v1.21.x salloc gave up: salloc: job 1458051 queued and waiting for resources → salloc: error: Unable to allocate resources: Connection timed out.

The build and all actual NIXL EP tests passed. The only failing branch waited a full hour for a free gb200nvl72_cx8 GPU node and never got one (the cluster partition was saturated / no nodes freed within the immediate window).

Summary: The ucx-v1.21.x parallel branch of nixl-ci-dl-gpu-ep #79 failed because its SLURM salloc could not obtain a GPU node within the 3600s immediate window; the actual build and EP elastic tests all passed.

Root cause: SLURM resource-allocation timeout — salloc ... --immediate=3600 for partition gb200nvl72_cx8 returned error: Unable to allocate resources: Connection timed out after ~60 minutes of job 1458051 queued and waiting for resources. The gb200nvl72_cx8 partition had no free node for the v1.21.x branch (the master branch consumed the only allocated node, job 1458049 on gb-nvl-118-compute07, until 12:08). This is an infrastructure/scheduling scarcity issue, not a code or test defect. The build itself (UCX + NIXL + nixlbench) compiled and installed cleanly, and every EP elastic test that did run (no_expansion.json, expansion_fault_contraction.json, with and without --disable-ll-nvlink) passed.

Note: this is NOT a hang — the process was legitimately blocked in the SLURM queue for the full, explicitly-configured --immediate=3600 window and then returned a scheduler error; there is no application log line that stalled.

Implicated commit: unknown — not code-related (commit d54a224 built and tested successfully on the master-UCX track).

File: SLURM allocation step in the Jenkins pipeline (slurm.allocation / the salloc -N 1 -p gb200nvl72_cx8 --immediate=3600 --time=01:30:00 --account=blackwell invocation for job name nixl-ci-ep-v1.21.x-79).

Suggested fix: Retry the build — this is a transient cluster-capacity failure. To reduce recurrence: (1) serialize or stagger the two UCX-variant EP branches so they don't both contend for a single gb200nvl72_cx8 node at once, or (2) request the two allocations up front / reuse one allocation across branches, and/or (3) add automatic re-queue-on-Unable to allocate resources retry logic to the slurm.allocation helper so a saturated partition doesn't fail the whole pipeline. No source code change is warranted.

Related: none found.

[brminich]

/build

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu-ep build 81…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit d54a224d

Notably, PR #1882 (the triggering PR) is titled "docs: refresh tracing.md after NVTX completeness + correlation" — a documentation-only change. This strongly confirms the failure is unrelated to the PR's code content; it's a pure infrastructure/scheduling failure.

The build phase succeeded entirely. The failure is purely in the Slurm allocation step waiting for GB200 nodes.

Summary: The nixl-ci-dl-gpu-ep build completed successfully; the job failed in the "Allocate DL EP Environment" stage when salloc could not obtain a GB200 compute node within the 1-hour immediate window.

Root cause: Slurm resource-allocation timeout, not an application hang or code defect. Both parallel branches issued salloc ... -p gb200nvl72_cx8 --immediate=3600 --account=blackwell and, after the job was queued and waited the full 3600 s for free nodes, Slurm returned error: Unable to allocate resources: Connection timed out. The ~60-minute silent gap (13:18:40 → 14:18:51 for v1.21.x; 13:19:22 → 14:19:31 for master) exactly matches the --immediate=3600 limit — this is Slurm intentionally giving up because the gb200nvl72_cx8 partition had no available nodes, i.e. an environment/capacity issue on the DL cluster, not a hung process. The triggering change (PR #1882) is documentation-only, so it cannot be the cause.

Implicated commit: none (infrastructure/capacity failure; PR #1882 d54a224 is docs-only and unrelated)

File: Jenkinsfile "Allocate DL EP Environment" stage — slurm.allocation call (partition: gb200nvl72_cx8, immediateTimeout: 3600, --account=blackwell)

Suggested fix: This is a transient cluster-capacity problem — retry the job when GB200 (gb200nvl72_cx8) nodes are available. If these timeouts are recurring, consider: (1) raising/monitoring --immediate handling so the pipeline reports "cluster busy" distinctly from a real failure, (2) adding automatic requeue/backoff on Unable to allocate resources, or (3) working with the cluster admins on partition capacity/reservation for the blackwell account. Do not treat this as a code regression in PR #1882.

Related: PR #1882 (triggering PR, docs-only); no matching issue for the allocation timeout was found.