[sethforprivacy]

Two silent-failure classes found while bringing up a second 2× GB10 pair from this recipe.

1. wrun() needs ssh -n

wrun() inherits the caller's stdin, so ssh consumes it. Any script that calls it from inside a heredoc-fed bash -s has the remainder of the script silently eaten: the worker step succeeds, every later line vanishes, exit status is 0, and nothing is logged.

./stop.sh is the everyday case. Driving it remotely, the worker stops and then every command after it is silently discarded. That cost us a genuinely confusing debugging session — the pair was down and our follow-up steps had simply never run.

The fix is one flag, but placement matters: -n goes at the wrun() call site, not in SSH_OPTS, because scp and rsync -e share SSH_OPTS and -n would break rsync's transport.

2. The fabric preflight warns where it should assert

preflight pinged the worker's CX7 IP at default size and only warned on failure. Three gaps:

• MTU is never verified. RoCE derives its IB MTU from the netdev MTU, so a single end left at 1500 caps the entire TP group — and a 56-byte ping cannot see it. Worth catching here specifically because the README already tells operators MTU 9000 is required; the check just wasn't there.
• An unreachable fabric peer only warned, so the run continued into a rendezvous hang rather than failing fast with a clear cause.
• *_CX7_IB was never checked against ibv_devices. A wrong device name doesn't error; NCCL degrades quietly.

This adds an -M do -s 8972 ping end-to-end so 9000 is proven in the path rather than read off one side, the netdev MTU read on both nodes, the configured RoCE device asserted present in ibv_devices on both nodes, and error/exit 1 instead of warn when the fabric peer is unreachable.

Validation

On a correctly configured 2× GB10 pair (200 Gb/s DAC, MTU 9000 both ends, rocep1s0f0):

PASS: 8972B unfragmented ping ok
  head mtu(enp1s0f0np0)=9000
  worker mtu(enp1s0f0np0)=9000
PASS: head sees rocep1s0f0
PASS: worker sees rocep1s0f0
PASS: 20000B correctly rejected (assert has teeth)

The last line is the negative control — the MTU assert fails when it should, rather than passing vacuously. No false positive on healthy hardware. bash -n start.sh clean.

[MiaAI-Lab]

Reviewed and merging.

Both halves are real bugs:

1. ssh -n belongs at the wrun() call site, not in SSH_OPTS (scp/rsync share those opts). Without it, a heredoc-fed remote script silently eats everything after the first wrun — ./stop.sh is the everyday case.

2. Fail-fast fabric preflight is the right default for this recipe. A warn-and-continue on an unreachable CX7 peer, no jumbo MTU check, and no ibv_devices assert were all silent paths into rendezvous hangs / NCCL degradation.

Nit, not blocking for the default single-device Spark pair: ibv_devices | grep -qw "${HEAD_CX7_IB}" treats the documented comma-separated dual-rail value (HEAD_CX7_IB="rocep1s0f1,roceP2p1s0f1") as one word and will false-fail on that config. Split on comma in a follow-up if you use the second rail.

Behavior change to be aware of: a misconfigured fabric now refuses to start instead of warning and hanging. That is intended. Jumbo ICMP can theoretically fail while RoCE still works; unlikely on a DAC CX7 pair.