nixlbench: limit Neuron core allocation in SG mode for multi-processing - #1913
Conversation
|
👋 Hi fengjica! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis change updates ChangesNeuron core allocation changes
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
c42c358 to
74a44d8
Compare
|
/build |
|
👀 Investigating |
|
🤖 CI Triage Agent — I have clear evidence now. The build stages succeeded; the failing stages (183 and 200 - "Allocate DL EP Environment") failed at Slurm resource allocation. Let me confirm the timing to distinguish hang vs. genuine wait. Looking at the log for stage 200:
That's exactly ~1 hour (3600s) between the salloc invocation and the timeout error. The This is not a hang in nixl code, and not a code defect at all. The build of NIXL + UCX + nixlbench all completed successfully (stages 159, 162 succeeded). Both allocation attempts (v1.22.x → job 1507450, and master → job 1507420) sat in the Slurm queue for the full immediate-timeout window on partition The earlier stage failures (124/125 Summary: The Root cause: Infrastructure/capacity, not a code defect. Both allocation attempts ran Implicated commit: unknown — not code-related. The build for commit [REDACTED:Hex High Entropy String] compiled cleanly; failure is in cluster resource allocation. File: The Jenkins pipeline SLURM allocation step ( Suggested fix:
Related: none found (no code issue to correlate; failure is Slurm allocation
|
|
🤖 CI Triage Agent — TL;DR: The "Clang Format Check" failed because Full analysisSummary: GitHub Actions Root cause: The newly added Implicated commit: [REDACTED:Hex High Entropy String] (PR #1913, branch File: benchmark/nixlbench/src/utils/neuron.cpp (the Suggested fix: Reformat the file locally and commit the result: The expected result matches what CI proposed: std::cerr << "nixlbench: NEURON_RT_VISIBLE_CORES is set to \"" << visible_cores
<< "\" but SG mode uses only the first core (vnc=0)" << std::endl;Then Related: none
|
48faf8b to
b618855
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmark/nixlbench/src/utils/neuron.cpp`:
- Around line 160-163: Run clang-format-19 in place on the file containing the
NEURON_RT_VISIBLE_CORES warning, ensuring the stream expression in the neuron
utility is reformatted to the repository’s expected layout. Commit the resulting
formatting-only change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 3f2b97d6-75de-4d70-9c42-56bc9af322b4
📒 Files selected for processing (1)
benchmark/nixlbench/src/utils/neuron.cpp
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmark/nixlbench/src/utils/neuron.cpp`:
- Around line 155-158: Check the return value of setenv in the SG-mode branch
before printing the success message or proceeding to nrt_init. On failure,
report the error and abort initialization; only log that NEURON_RT_NUM_CORES was
set when setenv succeeds.
- Around line 152-165: Update the SG core-handling condition in neuron.cpp to
use IS_PAIRWISE_AND_SG() instead of checking xferBenchConfig::mode alone. Keep
the existing environment-variable setup and warning behavior unchanged, but
ensure it runs only for pairwise SG configurations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 98fba73b-3d01-4abe-9c5e-d964c60760c8
📒 Files selected for processing (1)
benchmark/nixlbench/src/utils/neuron.cpp
b618855 to
da85dd0
Compare
|
🤖 CI Triage Agent — TL;DR: The Clang Format Check failed because Full analysisSummary: Root cause: The PR (#1913) modified std::cerr << "nixlbench: SG mode — set NEURON_RT_NUM_CORES=1"
<< std::endl;to a single line ending in Implicated commit: [REDACTED:Hex High Entropy String] (PR #1913, branch File: Suggested fix: Run the repository's formatter locally on the changed file and commit the result: Concretely, collapse the wrapped Related: none found.
|
da85dd0 to
e7a3114
Compare
e7a3114 to
efd9672
Compare
|
@oa-aws Could you look again? The diff from the last revision is in the second commit |
On Neuron instances, nrt_init() claims all visible Neuron cores by default. When testing multiple nixlbench processes in SG pairwise mode (one per device), each process tries to claim all cores, causing "Logical Neuron Core(s) not available" failures for all but the first process. Set NEURON_RT_NUM_CORES=1 before nrt_init() in SG mode so each process only reserves one core. MG mode is unaffected (single process per host, claims all cores).
16d5ac4 to
d5f5b14
Compare
|
Squashed commits into one, and then rebased to head as there will be merge conflict. |
|
Hi @aranadive @brminich @ovidiusm would you mind taking a look at this PR? The delta from the last revision where @oa-aws approved is only that of rebasing to head and merge. |
|
/ok to test 86a0413 |
|
/build |
|
🤖 CI Triage Agent — TL;DR: The SGLang sanity stage failed because the prefill sglang server was scheduled onto a GPU that already had most of its memory occupied (only 16.41 GB free vs. 183.44 GB on the decode GPU), so it crashed at init with Full analysisSummary: Stage 580 "Run SGLang sanity" failed — the prefill sglang server OOM-crashed on startup and the health check timed out after 300s. (Stage 333's FAILURE is only parallel-branch abort propagation; the wheel build itself, stage 419, succeeded.) Root cause: The prefill server ( Implicated commit: unknown — no code regression; environmental. Recent touches to File: Suggested fix: Two-part: (1) Make the launch robust to a dirty GPU — before allocating, assert the target GPUs are near-idle (e.g. Related: PRs #1995 (router port fix) and #1777 (added this sanity test); none address the memory-contention failure mode.
|
|
🤖 CI Triage Agent — TL;DR: The "Allocate DL EP Environment" stage failed because the Slurm Full analysisSummary: GPU environment allocation on the dlcluster Slurm Root cause: Implicated commit: unknown (not a code regression; commit 5be5a1b builds cleanly through image push) File: N/A — failure is in the Jenkins Slurm allocation step ( Suggested fix: Re-run build #519 once Related: none |
|
/ok to test 8d1c007 |
|
/build |
|
🤖 CI Triage Agent — TL;DR: The vLLM sanity stage failed because both vLLM engine cores aborted at startup with Full analysisSummary: Stage "Run vLLM sanity" (#559) failed — vLLM prefill and decode engine cores could not initialize because insufficient free GPU memory was available; the Root cause: On the shared GB200 node Implicated commit: unknown — not a code regression; infrastructure/resource contention. File: Suggested fix: Re-run the build; the allocation should land on a clean node. To harden against recurrence: (1) before launching the servers, assert each target GPU is mostly free (e.g. Related: none found. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage 200 "Allocate DL EP Environment" failed: Slurm could not allocate a GB200 node within the immediate-wait window. Root cause: Implicated commit: none — not caused by commit [REDACTED:Hex High Entropy String]; this is a cluster/infrastructure resource-allocation failure. File: N/A (Jenkins pipeline Slurm allocation step, partition Suggested fix: Re-run the build once GB200 ( Related: none found.
|
|
🤖 CI Triage Agent — TL;DR: The "Run Nixlbench tests" stage hung for ~22 minutes after the last UCCL ASIO transfer test ("Engine destroyed" at 00:29:41) until Jenkins killed it with SIGTERM (exit 143); this is the known UCCL nixlbench hang (issue #1999), and the fix is to skip the UCCL nixlbench tests (as PR #2000 began, but the ASIO UCCL block must also be disabled). Full analysisSummary: Root cause: A hang (not a timeout) in the UCCL backend during/after the ASIO transfer tests. The largest inter-line gap is ~22 minutes of total silence immediately after the final UCCL ASIO test ( Implicated commit: Test harness ordering/skip issue rather than a single regression; workaround commit File: Suggested fix: Also skip/guard the UCCL ASIO nixlbench loop at lines 95-103 (mirror the commented-out ETCD block at 122-132, referencing issue #1999) until the UCCL hang is fixed. Re-run PR #1913 on top of the branch that includes #2000. As a defensive measure, wrap each Related: Issue #1999 (UCCL nixlbench test hangs), PR #2000 (CI: Skip UCCL nixlbench tests), PR #1724 (UCCL crash fix + GDB trap). |
|
🤖 CI Triage Agent — TL;DR: The "Run Nixlbench tests" stage crashed with exit code 139 (SIGSEGV) in the UCCL backend's engine teardown during the ASIO Full analysisSummary: Root cause: A SIGSEGV in the UCCL backend. The benchmark printed its full results row ( Implicated commit: UCCL backend by Pravein Govindan Kannan — most recently File: Suggested fix: Skip/comment out the ASIO-runtime UCCL nixlbench loop (lines 95-103) the same way the ETCD UCCL loop (lines 122-132) was disabled, referencing issue #1999, until the UCCL segfault-on-teardown is root-caused. Longer term, investigate the UCCL engine destructor for a double-free/use-after-free (the crash happens right after "Engine destroyed") and enable core-dump analysis per Related: Issue #1999 "UCCL nixlbench test hangs" (#1999); PRs #1428, #1271, #1151, #895 touching the UCCL backend.
|
6 participants
On Neuron instances, nrt_init() claims all visible Neuron cores by default. When testing multiple nixlbench processes in SG pairwise mode (one process per device), each process tries to claim all cores, causing "Logical Neuron Core(s) not available" failures for all but the first process.
Set NEURON_RT_NUM_CORES=1 before nrt_init() in SG mode so each process only reserves one core.
MG mode is unaffected (single process per host, claims all cores).
What?
Fix nixlbench multiprocessing with neruon runtime.
Why?
Without the fix, nixlbench multiprocessing (SG mode) would fail.
How?
Set NEURON_RT_NUM_CORES = 1, before nrt_init().
Tested with trn2.48xlarge instance.
Summary by CodeRabbit