[yafshar]

Problem

When the trainer and the weight consumer (orchestrator or evals process) run on different hosts and share the output dir over NFS, a weight-broadcast handshake marker can take ~30 s to become visible to the side waiting for it.

The NFS client caches negative lookups. If the waiter's Path.exists() runs just before the other host creates the marker, the client keeps answering "not found" from its cache until the parent directory's cached attributes expire (acdirmin, 30 s by default). Every broadcast has two cross-host waits: the consumer waits for .sender_ready, and the trainer waits for .receiver_ready. In one split deployment, the weights moved in under 1 s, but every broadcast took ~31 s, about a third of the step time.

Fix

marker_exists in transports/weights/base.py opens the marker's step dir, and the broadcast dir above it, before calling marker.exists(). Opening a directory makes the NFS client revalidate it (close-to-open consistency), so a stale "not found" is dropped once the directory has changed. The broadcast dir is opened too because the step dir may not exist yet when the consumer starts polling, since the trainer recreates it for every broadcast.

It replaces the bare .exists() in _wait_for_receiver_ready, is_published and wait_published. The added cost is up to two directory opens per marker check.

Measured

The setup was two hosts sharing an NFSv3 mount with default options. One host creates a marker after a random 0.2–2 s delay, and the other polls for it at 0.1 s, as the handshake does. Each leg is timed on the creating host's monotonic clock, which stops when the poller reports the marker back over TCP, so clock skew doesn't affect the numbers. Every trial uses a fresh step dir. The trials cover both directions and both layouts: the waiter created the step dir (as for .receiver_ready), or the creator made the dir and the marker together (as for .sender_ready).

Polling method	Trials	Median (range across groups)	Worst
marker.exists()	20	28.4–29.3 s	29.7 s
marker_exists(marker)	20	0.54–0.64 s	0.83 s

Plain polling stalled in every direction and layout, each time until ~30 s after the first poll.

Scope

This PR covers the handshake markers only. wait_for_path and sync_wait_for_path in utils/pathing.py, which serve the filesystem weight transport's .finished wait and the filesystem batch transport, still use a bare .exists(). They have the same exposure across hosts on NFS. Deployments that can remount have another option: the lookupcache=positive mount option disables negative lookup caching.

[yafshar]

Ready for review. When the trainer and the consumer share the output dir over NFS, each handshake marker could take ~30 s to become visible, because the NFS client caches "not found" answers. This revalidates the directory before each check, which cuts the wait from ~29 s to ~0.6 s; the measurements are in the description. The utils/pathing.py waits are left out of scope, as noted.

@biswapanda could you review? @samsja @faresobeid for approval.