[sethforprivacy]

cmd_serve decides the pair is already running by looking at the head container only:

if docker ps --format '{{.Names}}' | grep -qx "${HEAD_CONTAINER}"; then
  ok "${HEAD_CONTAINER} already running"
  curl -fsS "${AUTH_CURL[@]}" "${READY_URL}" >/dev/null 2>&1 && ok "API is ready: ${READY_URL}" || info "API not ready yet (still booting?)"
  info "logs: ./start.sh logs   ·   stop: ./start.sh stop"
  exit 0
fi

A TP=2 group isn't running because rank 0 is. With the head up and the worker gone, rank 0 waits in NCCL rendezvous indefinitely — or worse, against a stale rank 1 — and this path prints already running, downgrades the failed readiness probe to an advisory API not ready yet (still booting?), and exits 0. The curl can't rescue it: its result is discarded on both branches.

How we hit it

A head node needed a power cycle. Its container carried a restart policy, so on boot Docker restored rank 0 unordered, against a leftover rank 1 still up on the worker from before the outage. The pair sat at Init torch distributed begin and would have waited forever — and ./start.sh reported success on it. Recovery was: kill the worker container by hand, then ./start.sh stop && ./start.sh to get the worker-first ordering the script normally does so carefully.

Worth noting the ordering point generally: dockerd cannot order two nodes, so worker-first is only expressible from the head, in this script. That makes this check the only place the invariant can be enforced.

The change

Both ranks must be present to claim "already running". Head up + worker absent is reported as a half-assembled pair, with the recovery command, exit 1.

It deliberately does not auto-repair. Tearing down a head that may still be serving is the operator's call, and stop then start is already the documented flow.

Validation

On a live 2× GB10 pair:

• Both ranks up — unchanged: already running, API is ready, exit 0.
• Half-assembled branch — exercised by pointing WORKER_CONTAINER at a nonexistent name, so no container was touched: all four error lines print, exit is 1, and the head stayed up and serving throughout (/v1/models 200, restarts=0). This seemed better than deliberately dismantling a serving pair to prove a branch.
• The worker predicate checked both directions — true for the real container name, false for a bogus one.

bash -n start.sh clean. Branched from main independently of #13/#14/#15/#16; touches only the cmd_serve region.

[MiaAI-Lab]

Reviewed and merging.

A TP=2 group is not running because rank 0 is. The old cmd_serve short-circuit looked at the head container only, discarded the readiness curl, and exited 0 — including the half-assembled case (head up, worker gone) that sits in NCCL rendezvous forever. The power-cycle + restart-policy story is exactly how that happens.

Requiring both ranks to claim “already running”, reporting half-assembled with ./start.sh stop && ./start.sh, and not auto-repairing is the right call. Tearing down a head that may still be serving is the operator’s decision.

The inverse (worker up, head down) correctly falls through into the normal worker-first launch. Isolated to the cmd_serve region.