Skip to content

[CI] Check B200 NUMA mapping against sysfs numa_node - #40055

Merged
Fridge003 merged 1 commit into
mainfrom
mmangkad/b200-numa-sysfs-check
Sep 18, 2026
Merged

Fridge003 merged 1 commit into
mainfrom
mmangkad/b200-numa-sysfs-check

Conversation

@mmangkad

@mmangkad mmangkad commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

TestB200NumaTopology hardcoded a one-NUMA-node layout. Cirrascale B200 hosts give each PCIe root complex its own node, so the 4 visible GPUs span two:

AssertionError: 2 != 1 : Expected all visible 4-GPU B200 devices on one NUMA node, got {3, 5}

Failing job

before after
asserts all 4 GPUs on one node NVML affinity contains the GPU's sysfs numa_node
oracle hardcoded /sys/bus/pci/devices/<bdf>/numa_node
BDF from n/a torch.cuda, not NVML
BIOS-independent no yes

NVML is avoided for the BDF because resolving that handle needs _get_nvml_device_index, whose device_id fallback is one of the faults this test detects.

Host layout (cirrascale1 and cirrascale3 identical, 6 NUMA nodes):

GPUs NUMA node
0,1 0
2,3 2
4,5 3
6,7 5

Still fails on: device index returned instead of node, bitmask decode offset.

Verification

Test run in each B200 CI container, 4 GPUs compared each, no skips:

container vendor GPU -> NUMA node result
b200-cirrascale1-0123 Cirrascale 0,0,2,2 Ran 1 test ... OK
b200-cirrascale1-4567 Cirrascale 3,3,5,5 Ran 1 test ... OK
b200-cirrascale3-0123 Cirrascale 0,0,2,2 Ran 1 test ... OK
b200-cirrascale3-4567 Cirrascale 3,3,5,5 Ran 1 test ... OK
b200-di01-0123 DeepInfra 0,0,2,2 Ran 1 test ... OK
b200-di01-4567 DeepInfra 3,3,5,5 Ran 1 test ... OK
b200-fin03-4-0123 Verda 0,0,0,0 Ran 1 test ... OK
b200-fin03-4-4567 Verda 1,1,1,1 Ran 1 test ... OK

The Verda hosts have 2 NUMA nodes and are the layout the old assert was written for; they stay green.

@Fridge003
Fridge003 enabled auto-merge (squash) September 18, 2026 04:14
@Fridge003
Fridge003 disabled auto-merge September 18, 2026 04:14
@Fridge003
Fridge003 merged commit c055dc6 into main Sep 18, 2026
184 of 239 checks passed
@Fridge003
Fridge003 deleted the mmangkad/b200-numa-sysfs-check branch September 18, 2026 04:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants