Add periodic GPU sniff tests to detect hardware stragglers by deepakn94 · Pull Request #4662 · NVIDIA/Megatron-LM

deepakn94 · 2026-05-06T23:45:51Z

Summary

Add periodic GPU performance sniff tests to detect hardware degradation and stragglers during training. We will push all the collected data to a metrics service in a follow-up PR.

Motivation

Large-scale training runs are vulnerable to silent GPU/network degradation — a single slow node can bottleneck the entire job. This feature provides lightweight, periodic micro-benchmarks that flag outlier ranks by name and node, making it easy to identify and replace bad hardware without waiting for training throughput to visibly drop.

What it does

Adds megatron/training/gpu_sniff_test.py with five micro-benchmarks, each run at two message sizes (256 MiB and 1 MiB):
1. GEMMs — standard shapes + FFN shapes derived from training args (TFLOP/s/GPU).
2. All-reduce over global PG (bus bandwidth).
3. Reduce-scatter over TP PG (bus bandwidth).
4. All-to-all over EP PG (bus bandwidth).
5. Pairwise send/recv at multiple strides within DP PG (bus bandwidth).
Controlled by --gpu-sniff-test-interval N: runs once before training starts and every N iterations thereafter.
Outlier detection: any rank whose metric deviates from the mean by more than one standard deviation (and >10% of mean) is flagged with rank number and node hostname.
Can also be run standalone (no Megatron machinery needed) for cluster validation before launching training.
Accepts a ProcessGroupCollection to avoid relying on global parallel state.
Takes ~670ms per invocation (steady-state) on 32 GPUs.

Sample output (standalone, 16 nodes / 128x H100, EP=8, TP=4)

============================================================
  GPU sniff test -- standalone
============================================================
  GEMM throughput (8192x8192x8192, bf16) [TFLOP/s/GPU]: mean=798.70, min=767.89, max=830.25
  GEMM throughput (4096x4096x16384, bf16) [TFLOP/s/GPU]: mean=787.16, min=728.13, max=849.88
  All-reduce busbw (global PG, size=128, 268 MB) [GB/s]: mean=295.68, min=295.41, max=301.15
  All-reduce busbw (global PG, size=128, 1 MB) [GB/s]: mean=12.75, min=12.70, max=13.22
  Reduce-scatter busbw (TP PG, size=4, 268 MB) [GB/s]: mean=296.52, min=277.95, max=298.26
  Reduce-scatter busbw (TP PG, size=4, 1 MB) [GB/s]: mean=36.08, min=23.87, max=47.33
    OUTLIER rank 44 (pool0-00795): 23.95 (-33.6%)
    OUTLIER rank 45 (pool0-00795): 23.87 (-33.9%)
    OUTLIER rank 52 (pool0-00839): 47.33 (+31.2%)
    OUTLIER rank 53 (pool0-00839): 46.55 (+29.0%)
  All-to-all busbw (EP PG, size=8, 268 MB) [GB/s]: mean=311.27, min=306.46, max=313.27
  All-to-all busbw (EP PG, size=8, 1 MB) [GB/s]: mean=34.89, min=26.37, max=46.73
  Send/recv busbw (DP PG, 268 MB, e.g., rank 0 <-> rank 8) [GB/s]: mean=14.16, min=13.84, max=14.38
  Send/recv busbw (DP PG, 268 MB, e.g., rank 0 <-> rank 32) [GB/s]: mean=18.55, min=15.91, max=19.28
    OUTLIER rank 30 (pool0-00782): 15.91 (-14.2%)
    OUTLIER rank 62 (pool0-00854): 15.91 (-14.2%)
  Send/recv busbw (DP PG, 268 MB, e.g., rank 0 <-> rank 64) [GB/s]: mean=14.05, min=13.84, max=14.31
  Send/recv busbw (DP PG, 1 MB, e.g., rank 0 <-> rank 8) [GB/s]: mean=11.19, min=10.61, max=11.55
  Send/recv busbw (DP PG, 1 MB, e.g., rank 0 <-> rank 32) [GB/s]: mean=11.39, min=9.03, max=12.30
    OUTLIER rank 89 (pool0-00861): 9.13 (-19.9%)
    OUTLIER rank 121 (pool0-01641): 9.03 (-20.7%)
  Send/recv busbw (DP PG, 1 MB, e.g., rank 0 <-> rank 64) [GB/s]: mean=10.70, min=10.45, max=10.99
  Result: OUTLIERS DETECTED
============================================================

Sample output (integrated, 4 nodes / 32x H100, 8B model, TP=4)

[running GPU sniff test (before training)] datetime: 2026-05-07 11:03:14.131838
============================================================
  GPU sniff test -- before training
============================================================
  GEMM throughput (8192x8192x8192, bf16) [TFLOP/s/GPU]: mean=797.20, min=782.95, max=830.74
  GEMM throughput (4096x4096x16384, bf16) [TFLOP/s/GPU]: mean=797.78, min=730.94, max=834.51
  GEMM throughput (8192x5376x4096, bf16, up-proj/fc1) [TFLOP/s/GPU]: mean=704.02, min=650.62, max=751.67
  GEMM throughput (8192x4096x5376, bf16, down-proj/fc2) [TFLOP/s/GPU]: mean=759.92, min=709.86, max=803.85
  All-reduce busbw (global PG, size=32, 268 MB) [GB/s]: mean=245.15, min=244.14, max=246.10
  All-reduce busbw (global PG, size=32, 1 MB) [GB/s]: mean=6.04, min=6.03, max=6.04
  Reduce-scatter busbw (TP PG, size=4, 268 MB) [GB/s]: mean=296.77, min=296.07, max=297.55
  Reduce-scatter busbw (TP PG, size=4, 1 MB) [GB/s]: mean=20.44, min=19.14, max=21.38
  Send/recv busbw (DP PG, 268 MB, e.g., rank 0 <-> rank 4) [GB/s]: mean=44.07, min=44.02, max=44.11
  Send/recv busbw (DP PG, 268 MB, e.g., rank 0 <-> rank 8) [GB/s]: mean=43.49, min=43.24, max=43.72
  Send/recv busbw (DP PG, 268 MB, e.g., rank 0 <-> rank 16) [GB/s]: mean=39.99, min=39.82, max=40.15
  Send/recv busbw (DP PG, 1 MB, e.g., rank 0 <-> rank 4) [GB/s]: mean=15.84, min=14.37, max=16.72
  Send/recv busbw (DP PG, 1 MB, e.g., rank 0 <-> rank 8) [GB/s]: mean=10.49, min=10.14, max=10.87
  Send/recv busbw (DP PG, 1 MB, e.g., rank 0 <-> rank 16) [GB/s]: mean=10.46, min=10.10, max=10.76
  Result: ALL RANKS OK
============================================================
[finished GPU sniff test (before training)] datetime: 2026-05-07 11:03:16.162628

Standalone usage

torchrun --nproc_per_node=NUM_GPUS megatron/training/gpu_sniff_test.py \
  [--ep-size N] [--tp-size N] [--gemm-shapes MxNxK ...] \
  [--skip-gemm] [--skip-allreduce] [--skip-reducescatter] [--skip-alltoall] [--skip-sendrecv]

Test plan

Tested standalone on 1-node (4x GB300, 8x H100).
Tested standalone on 2-node (16x H100, EP=8).
Tested standalone on 4-node (32x H100, EP=8) — verified multi-stride send/recv.
Tested standalone on 16-node (128x H100, EP=8, TP=4) — all 5 benchmarks at both message sizes, hostname logging verified.
Tested standalone on 32-node (128x GB300, EP=64, segment=16).
Tested integrated mode on 4-node (32x H100, 8B model, TP=4) — sniff test fires before training and every 50 iterations (~670ms steady-state), hostname logging verified.

🤖 Generated with Claude Code

copy-pr-bot · 2026-05-06T23:45:55Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

asolergi-nv

2 small comments !

asolergi-nv · 2026-05-07T18:42:50Z

+    tp = args.tensor_model_parallel_size
+    gated = getattr(args, 'gated_linear_unit', False) or getattr(args, 'swiglu', False)
+
+    tokens = args.micro_batch_size * args.seq_length


Should we consider CP here too?

Let's see how well this does first. We can do CP later, but cross-node issues should be caught by the DP tests.

asolergi-nv · 2026-05-07T18:45:56Z

+# Send/recv benchmark.
+# ---------------------------------------------------------------------------
+
+def bench_sendrecv(group):


Should we protect this test when DP size is not a power of 2? Or switch to a ring based grouping?

The only problem with non-powers of 2 is some ranks might not participate in this send-recv pair. Will remove these from the report.

chtruong814 · 2026-05-07T23:52:29Z

/ok to test 0beca0e

svcnvidia-nemo-ci · 2026-05-08T00:37:32Z

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25529908995

Add a lightweight GPU benchmarking module that can be run periodically during training (controlled by --gpu-sniff-test-interval) or standalone. Three micro-benchmarks are run and metrics are gathered across all ranks: 1. GEMMs (standard shapes + FFN shapes from training args). 2. All-to-all over EP process group. 3. Pairwise send/recv at multiple strides within DP process group. Any rank whose throughput deviates from the mean by more than one standard deviation (and more than 10% of the mean) is flagged as an outlier. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

svcnvidia-nemo-ci · 2026-05-08T03:55:36Z

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25535780334

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch 5 times, most recently from f06fc00 to ff841df Compare May 7, 2026 04:51

deepakn94 marked this pull request as ready for review May 7, 2026 05:12

deepakn94 requested review from a team as code owners May 7, 2026 05:12

svcnvidia-nemo-ci requested a review from a team May 7, 2026 05:12

copy-pr-bot Bot temporarily deployed to test May 7, 2026 05:13 Inactive

svcnvidia-nemo-ci added the complexity: medium label May 7, 2026

jaredcasper reviewed May 7, 2026

View reviewed changes

Comment thread megatron/training/gpu_sniff_test.py Outdated

Comment thread megatron/training/gpu_sniff_test.py Outdated

Comment thread megatron/training/gpu_sniff_test.py Outdated

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch from ff841df to 5cc82c0 Compare May 7, 2026 16:52

copy-pr-bot Bot temporarily deployed to test May 7, 2026 16:53 Inactive

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch from 5cc82c0 to cabbf64 Compare May 7, 2026 17:35

copy-pr-bot Bot temporarily deployed to test May 7, 2026 17:37 Inactive

jaredcasper approved these changes May 7, 2026

View reviewed changes

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch from cabbf64 to fc37c4f Compare May 7, 2026 18:42

copy-pr-bot Bot temporarily deployed to test May 7, 2026 18:43 Inactive

asolergi-nv approved these changes May 7, 2026

View reviewed changes

svcnvidia-nemo-ci added the Approved All necessary approvals have been made label May 7, 2026

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch from fc37c4f to f6c873f Compare May 7, 2026 19:02

copy-pr-bot Bot temporarily deployed to test May 7, 2026 19:05 Inactive

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch from f6c873f to e939149 Compare May 7, 2026 20:51

copy-pr-bot Bot temporarily deployed to test May 7, 2026 20:52 Inactive

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch from e939149 to e7ab9e7 Compare May 7, 2026 21:21

copy-pr-bot Bot temporarily deployed to test May 7, 2026 21:23 Inactive

copy-pr-bot Bot temporarily deployed to test May 7, 2026 23:52 Inactive

deepakn94 changed the title ~~Add periodic GPU performance sniff tests~~ Add periodic GPU sniff tests to detect hardware stragglers May 8, 2026

deepakn94 enabled auto-merge May 8, 2026 00:33

deepakn94 added this pull request to the merge queue May 8, 2026

deepakn94 removed this pull request from the merge queue due to a manual request May 8, 2026

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch from 0beca0e to e41ac4c Compare May 8, 2026 00:48

copy-pr-bot Bot had a problem deploying to test May 8, 2026 00:49 Error

deepakn94 enabled auto-merge May 8, 2026 01:00

deepakn94 force-pushed the dnarayanan/gpu_sniff_test branch from e41ac4c to bab057d Compare May 8, 2026 01:01

copy-pr-bot Bot temporarily deployed to test May 8, 2026 01:02 Inactive

deepakn94 added this pull request to the merge queue May 8, 2026

Merged via the queue into NVIDIA:main with commit 932d9ee May 8, 2026
67 checks passed

deepakn94 deleted the dnarayanan/gpu_sniff_test branch May 8, 2026 05:07

yhgalaxy pushed a commit to yhgalaxy/Megatron-LM that referenced this pull request Jun 17, 2026

Add periodic GPU sniff tests to detect hardware stragglers (NVIDIA#4662)

97aa584

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>

terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request Aug 3, 2026

Add periodic GPU sniff tests to detect hardware stragglers (NVIDIA#4662)

6b134b3

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Navigation Menu

Uh oh!

Add periodic GPU sniff tests to detect hardware stragglers - #4662

Add periodic GPU sniff tests to detect hardware stragglers#4662
deepakn94 merged 1 commit into
NVIDIA:mainfrom
deepakn94:dnarayanan/gpu_sniff_test

deepakn94 commented May 6, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented May 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asolergi-nv left a comment

Uh oh!

asolergi-nv May 7, 2026

Uh oh!

deepakn94 May 7, 2026

Uh oh!

asolergi-nv May 7, 2026

Uh oh!

deepakn94 May 7, 2026

Uh oh!

chtruong814 commented May 7, 2026

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

deepakn94 commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

What it does

Sample output (standalone, 16 nodes / 128x H100, EP=8, TP=4)

Sample output (integrated, 4 nodes / 32x H100, 8B model, TP=4)

Standalone usage

Test plan

Uh oh!

copy-pr-bot Bot commented May 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asolergi-nv left a comment

Choose a reason for hiding this comment

Uh oh!

asolergi-nv May 7, 2026

Choose a reason for hiding this comment

Uh oh!

deepakn94 May 7, 2026

Choose a reason for hiding this comment

Uh oh!

asolergi-nv May 7, 2026

Choose a reason for hiding this comment

Uh oh!

deepakn94 May 7, 2026

Choose a reason for hiding this comment

Uh oh!

chtruong814 commented May 7, 2026

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

deepakn94 commented May 6, 2026 •

edited

Loading