Add multi-node sandbox support for SLURM clusters

[gwarmstrong]

Summary

• Adds multi-node sandbox support, enabling code execution workers to be distributed across multiple SLURM nodes with nginx load balancing and session affinity
• Auto-detects SLURM multi-node environments (via SLURM_JOB_NODELIST), expands compressed nodelists, and coordinates port assignments across nodes via shared filesystem
• Backward-compatible: single-node mode works identically when SLURM variables are absent

Changes

dockerfiles/sandbox/start-with-nginx.sh (major rewrite)

• SLURM nodelist expansion (supports node[001-016], gpu[01-02],cpu[01-03], etc.)
• TCP socket workers (replacing unix sockets) with configurable base port
• Cross-node port coordination via shared filesystem (/nemo_run/ or /workspace/)
• Master/worker node roles: master runs nginx LB with cross-node upstream, workers proxy to master
• Port conflict retry algorithm for TCP port binding failures
• Parallel health checks for both local and remote workers
• Network blocking support for all modes with transparent diagnostics
• Comprehensive environment variable documentation header
• SANDBOX_FORCE_SINGLE_NODE override for debugging

dockerfiles/sandbox/nginx-worker-proxy.conf.template (new file)

• Extracted nginx config template for worker nodes that proxy to master's LB

dockerfiles/Dockerfile.sandbox (minor)

• Move COPY start-with-nginx.sh after dependency layers for better Docker cache
• Use exec-form CMD ["/start-with-nginx.sh"]
• Add COPY for new nginx-worker-proxy.conf.template

dockerfiles/sandbox/nginx.conf.template (minor)

• Updated comments to document both single-node and multi-node upstream modes

Test plan

• Validated on DFW 16-node run (128 workers/node): 9594 successful requests, 0 5xx errors
• Validated on DFW 2-node run
• Single-node backward compatibility verified (no SLURM vars → same behavior as before)
• CI tests pass

Summary by CodeRabbit

• New Features

  • Multi-node sandboxing with coordinated worker discovery, session-aware routing, health/readiness checks, and session-affinity for stable routing.
  • TCP-based worker networking with proxying and an nginx status endpoint for monitoring.

• Chores

  • Optimized container startup ordering and command form for improved caching and startup behavior.
  • Added new proxy configuration template and introduced a configuration flag to span group nodes.

[greptile-apps]

_{1 file reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

useless cat with grep -q

Suggested change

	if [ -f "$port_file" ] && cat "$port_file" 2>/dev/null | grep -q "PORT_REPORT_COMPLETE"; then
	if [ -f "$port_file" ] && grep -q "PORT_REPORT_COMPLETE" "$port_file" 2>/dev/null; then

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[greptile-apps]

race condition: all nodes execute rm -f on shared filesystem simultaneously

If multiple nodes start at nearly the same time, they can race to delete each other's port files. A node that completes writing its ports could have its file deleted by another node still executing this line.

Suggested change

	rm -f "$PORTS_REPORT_DIR"/*_ports.txt 2>/dev/null || true
	rm -f "$PORTS_REPORT_DIR/${CURRENT_NODE_SHORT}_ports.txt" 2>/dev/null || true

[greptile-apps]

potential array index out of bounds if NUM_WORKERS is 0

If NUM_WORKERS=0 (either explicitly set or through nproc --all returning 0), this line would access ACTUAL_WORKER_PORTS[-1] which is invalid in bash.

Check NUM_WORKERS > 0 before array access or earlier in the script.

[greptile-apps]

same array bounds issue as line 655

Suggested change

	echo " Local workers: $NUM_WORKERS (ports ${ACTUAL_WORKER_PORTS[0]}-${ACTUAL_WORKER_PORTS[$((NUM_WORKERS-1))]})"
	echo " Local workers: $NUM_WORKERS (ports ${ACTUAL_WORKER_PORTS[0]:-none}-${ACTUAL_WORKER_PORTS[$((NUM_WORKERS>0?NUM_WORKERS-1:0))]:-none})"

[coderabbitai]

📝 Walkthrough

Walkthrough

Reorganizes the sandbox Dockerfile and adds templates, and substantially rewrites the startup script to support TCP-based uWSGI workers for single-node and SLURM-detected multi-node deployments with dynamic Nginx config generation, cross-node port coordination, remote health checks, and master/worker orchestration.

Changes

Cohort / File(s)	Summary
Dockerfile dockerfiles/Dockerfile.sandbox	Reordered COPY operations for cache optimization, added copy of nginx-worker-proxy.conf.template, and changed CMD from shell form to exec form (["/start-with-nginx.sh"]).
Nginx templates dockerfiles/sandbox/nginx.conf.template, dockerfiles/sandbox/nginx-worker-proxy.conf.template	Expanded upstream documentation and session-affinity notes in nginx.conf.template; added new nginx-worker-proxy.conf.template that proxies to a master load balancer with consistent-hash affinity, proxy headers, extended timeouts, disabled buffering, logging, and /nginx-status stub_status.
Startup script dockerfiles/sandbox/start-with-nginx.sh	Full rewrite to support multi-node deployments: SLURM nodelist expansion, master/worker role detection, per-worker TCP ports with retry/offset, shared port reporting directory, wait/verify aggregation of remote ports, dynamic nginx config generation & validation, parallel remote health checks, worker spawn/restart loops, optional network-blocking, and proxy behavior for non-master workers.
Python config nemo_skills/pipeline/utils/scripts.py	Added new public boolean field SandboxScript.span_group_nodes: bool = True to indicate sandbox spans group nodes (config flag; no behavior changes within file).

Sequence Diagram

sequenceDiagram
    participant SLURM as SLURM Cluster
    participant Master as Master Node<br/>(start-with-nginx.sh)
    participant Worker as Worker Node(s)<br/>(start-with-nginx.sh)
    participant Nginx as Nginx LB
    participant uWSGI as uWSGI Workers<br/>(TCP)
    participant PortDir as Shared Port<br/>Coordination Dir

    Master->>SLURM: Query node list (SLURM_NODELIST)
    SLURM-->>Master: Node hostnames & count
    Master->>Master: Determine master/worker role and init ports

    par Multi-Node Startup
        Master->>uWSGI: Start local uWSGI workers on BASE_PORT + offsets
        uWSGI-->>PortDir: Write per-node port report
        Worker->>uWSGI: Start local uWSGI workers on assigned ports
        Worker-->>PortDir: Write per-node port report
        Master->>Nginx: Generate & validate nginx.conf (template + collected ports)
        Master->>Nginx: Start/Reload Nginx with TCP upstreams
        Worker->>Nginx: Start local proxy to master (if non-master)
    end

    Master->>PortDir: wait_for_port_reports() and aggregate reports
    PortDir-->>Master: Collected ports from all nodes
    Master->>Master: Build upstreams with all worker endpoints
    Master->>Worker: verify_remote_workers() (parallel health checks)
    Worker-->>Master: Health responses

    Master->>Master: Monitor loop: health checks, restarts, nginx status
    Master->>Nginx: Monitor nginx and reload on config changes

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Add multi-node sandbox support for SLURM clusters' directly and clearly summarizes the main change: introducing multi-node sandbox capabilities for SLURM environments.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch georgea/multinode-sandbox-pr

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@dockerfiles/sandbox/start-with-nginx.sh`:
- Around line 335-337: The startup cleanup removes every node's port report
files (rm -f "$PORTS_REPORT_DIR"/*_ports.txt) causing a race where slow nodes
delete other nodes' reports; change the removal to only the current node's
report (e.g., rm -f "$PORTS_REPORT_DIR/${NODE_NAME}_ports.txt" or use hostname:
"${PORTS_REPORT_DIR}/$(hostname)_ports.txt") so other nodes' files remain;
update start-with-nginx.sh where PORTS_REPORT_DIR is used and ensure
wait_for_port_reports still reads all reports from the directory.

🧹 Nitpick comments (4)

dockerfiles/sandbox/start-with-nginx.sh (4)

117-147: Workers bind to 0.0.0.0 unconditionally — consider restricting in single-node mode.

Line 126 binds uWSGI workers to 0.0.0.0:${WORKER_PORT}, which is required for multi-node cross-node communication but unnecessarily exposes workers on all interfaces in single-node mode. If the container's network isn't fully isolated, any host on the network can reach individual workers directly (bypassing nginx session affinity).

Suggested conditional bind address

+    if [ "$NODE_COUNT" -gt 1 ]; then
+        BIND_ADDR="0.0.0.0"
+    else
+        BIND_ADDR="127.0.0.1"
+    fi
+
     cat > /tmp/worker${i}_uwsgi.ini << EOF
 [uwsgi]
 module = main
 callable = app
 processes = ${UWSGI_PROCESSES}
-http-socket = 0.0.0.0:${WORKER_PORT}
+http-socket = ${BIND_ADDR}:${WORKER_PORT}

Note: BIND_ADDR would need to be determined before start_worker_fast is called (e.g., set once after node discovery).

---

144-144: Shell redirections without commands (SC2188).

Lines 144, 560, 577, and 594 use bare > file to truncate files. While this works in bash, it's flagged by Shellcheck (SC2188) and can confuse readers. The idiomatic form is : > file (using the no-op builtin).

Example fix

-    > /var/log/worker${i}.log
+    : > /var/log/worker${i}.log

-    > "$PORTS_FILE"
+    : > "$PORTS_FILE"

-        > $UPSTREAM_FILE
+        : > $UPSTREAM_FILE

Also applies to: 560-560, 577-577, 594-594

---

247-261: Exported temp dir + xargs {} substitution — minor robustness notes.

1. export REMOTE_HEALTH_DIR (Line 245) leaks into the global environment. Since verify_remote_workers is called only once this is harmless, but local + explicit export or passing via env to xargs would be cleaner.
2. In the xargs -I {} sh -c 'endpoint="{}"' pattern (Line 254–261), if an endpoint string ever contains a single quote, it would break the sh -c quoting. This can't happen with valid host:port values, but a defensive alternative is to pipe through sh -c 'endpoint="$1"; ...' _ {} using positional args.

Both are low-risk given the trusted input, mentioning for hardening only.

---

668-705: Monitoring loop: restarted worker uses same port but nginx upstream is static.

The monitoring loop restarts crashed workers via start_worker which reuses the original port. This works because the restarted worker binds to the same port that nginx already has in its upstream — no config reload needed. This is a correct and practical design choice.

One edge consideration: if a restarted worker fails to bind (port still held by the dying process, or network blocking active), the monitoring loop will re-attempt every 10 seconds indefinitely. With network blocking, the warning is emitted (Lines 687–691). Without network blocking, the retry is silent after the first warning. Consider adding a retry counter or backoff to avoid log flooding for persistently-failing workers.

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

useless cat - xargs can read from file directly

Suggested change

	cat "$endpoints_file" | xargs -P 64 -I {} sh -c '
	xargs -P 64 -I {} sh -c '

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[Kipok]

please rebuild sandbox and run slurm tests. If those pass, we should be good to merge

[greptile-apps]

_{3 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (3)

nemo_skills/inference/generate.py
Missing key validation

STRUCTURED_OUTPUTS[self.cfg.structured_output] will raise a KeyError for any typo/unsupported value passed via ++structured_output=... (including from --extra_judge_args). Since this is user-facing CLI config, it should fail with a clear error listing valid keys rather than a raw KeyError stacktrace.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

---

tests/test_generation.py
Integration test hits external API

This test invokes ns eval against https://integrate.api.nvidia.com/v1 and real models. In CI (and for most contributors) this will deterministically fail due to missing credentials/network access, and it’s also very slow/flaky. Unit tests here should mock the model client / generation call (or be marked to skip unless explicit env vars are present).

---

.github/workflows/tests.yml
CI now prepares HLE dataset

ns prepare_data ... hle changes CI to download/prepare an additional dataset. If HLE requires credentials, large downloads, or is otherwise not reliably available in CI, this will break the workflow. Please confirm hle is lightweight/public like the other CI datasets, or gate it behind a condition/fixture used only in dedicated tests.

[coderabbitai]

Actionable comments posted: 3

🤖 Fix all issues with AI agents

In `@dockerfiles/sandbox/start-with-nginx.sh`:
- Around line 651-659: The port display is misleading because
ACTUAL_WORKER_PORTS may be non-contiguous after conflict retries; update the
echo logic in the IS_MASTER/WORKER blocks to either print the full list of
ACTUAL_WORKER_PORTS (iterating the ACTUAL_WORKER_PORTS array up to NUM_WORKERS)
or detect non-contiguity and append a "(non-contiguous ports: ...)" note;
specifically change the lines that currently emit
"${ACTUAL_WORKER_PORTS[0]}-${ACTUAL_WORKER_PORTS[$((NUM_WORKERS-1))]}" and the
similar WORKER echo to instead join and print each ACTUAL_WORKER_PORTS[i] (or
print the range plus a non-contiguous warning) so operators see the real ports
when retries/offsets occurred.
- Around line 685-694: When a worker dies and NETWORK_BLOCKING_ACTIVE=1, avoid
immediately restarting it to prevent an infinite futile restart loop; modify the
monitor logic around the kill-check and restart (the block that calls
start_worker and updates WORKER_PIDS and ACTUAL_WORKER_PORTS) to either skip
restarting when NETWORK_BLOCKING_ACTIVE is set or implement a per-worker retry
cap (e.g., track restart counts in an array like WORKER_RETRIES and stop
restarting after N attempts) and log a single clear message when restarts are
suppressed; ensure you reference NETWORK_BLOCKING_ACTIVE, start_worker,
WORKER_PIDS, ACTUAL_WORKER_PORTS (and add WORKER_RETRIES if using the cap) when
making the change so the monitor stops spawning doomed processes.
- Line 144: The script uses a bare redirection of the form >
/var/log/worker${i}.log which is not POSIX-portable; replace each bare
redirection of the worker log (the occurrences that use >
/var/log/worker${i}.log) with an explicit no-op redirection using the shell
builtin (i.e., use a ":" no-op followed by the redirection) so the intent is
explicit and shellcheck SC2188 is satisfied; apply the same change for the other
matching occurrences with the worker log filenames.

🧹 Nitpick comments (2)

dockerfiles/sandbox/start-with-nginx.sh (2)

254-261: xargs -I {} sh -c '…{}' is a shell-injection anti-pattern.

Although endpoints are internally generated, interpolating {} directly inside sh -c breaks if the value ever contains quotes or shell metacharacters. Pass the value as a positional parameter instead.

Proposed fix

-        cat "$endpoints_file" | xargs -P 64 -I {} sh -c '
-            endpoint="{}"
-            status_file="$REMOTE_HEALTH_DIR/$(echo "$endpoint" | tr ":" "_")"
+        cat "$endpoints_file" | xargs -P 64 -I {} sh -c '
+            endpoint="$1"
+            status_file="$REMOTE_HEALTH_DIR/$(echo "$endpoint" | tr ":" "_")"
             [ -f "$status_file" ] && exit 0
             if curl -s -f --connect-timeout 2 --max-time 5 "http://${endpoint}/health" > /dev/null 2>&1; then
                 touch "$status_file"
             fi
-        '
+        ' _ {}

---

668-679: Background stats subshell PID is not tracked by cleanup().

The monitoring subshell spawned here won't be killed by the cleanup trap. In a container this is benign (container stop kills all), but for completeness consider capturing the PID and adding it to the kill list.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Bare redirection without a command (SC2188).

> file works in bash but is not POSIX-portable and triggers shellcheck SC2188. Use : > to make intent explicit.

Proposed fix

-    > /var/log/worker${i}.log
+    : > /var/log/worker${i}.log

Same pattern on lines 560, 577, and 594.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	> /var/log/worker${i}.log
	: > /var/log/worker${i}.log

🧰 Tools

🪛 Shellcheck (0.11.0)

[warning] 144-144: This redirection doesn't have a command. Move to its command (or use 'true' as no-op).

(SC2188)

🤖 Prompt for AI Agents

In `@dockerfiles/sandbox/start-with-nginx.sh` at line 144, The script uses a bare
redirection of the form > /var/log/worker${i}.log which is not POSIX-portable;
replace each bare redirection of the worker log (the occurrences that use >
/var/log/worker${i}.log) with an explicit no-op redirection using the shell
builtin (i.e., use a ":" no-op followed by the redirection) so the intent is
explicit and shellcheck SC2188 is satisfied; apply the same change for the other
matching occurrences with the worker log filenames.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Port range display is misleading after conflict retries.

After the retry algorithm applies offsets (e.g., +200), ports are no longer contiguous. Displaying ${ACTUAL_WORKER_PORTS[0]}-${ACTUAL_WORKER_PORTS[$((NUM_WORKERS-1))]} implies a contiguous range, which could confuse operators. Consider listing all ports or noting "non-contiguous" when retries occurred.

🤖 Prompt for AI Agents

In `@dockerfiles/sandbox/start-with-nginx.sh` around lines 651 - 659, The port
display is misleading because ACTUAL_WORKER_PORTS may be non-contiguous after
conflict retries; update the echo logic in the IS_MASTER/WORKER blocks to either
print the full list of ACTUAL_WORKER_PORTS (iterating the ACTUAL_WORKER_PORTS
array up to NUM_WORKERS) or detect non-contiguity and append a "(non-contiguous
ports: ...)" note; specifically change the lines that currently emit
"${ACTUAL_WORKER_PORTS[0]}-${ACTUAL_WORKER_PORTS[$((NUM_WORKERS-1))]}" and the
similar WORKER echo to instead join and print each ACTUAL_WORKER_PORTS[i] (or
print the range plus a non-contiguous warning) so operators see the real ports
when retries/offsets occurred.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Futile restart loop when network blocking is active.

When NETWORK_BLOCKING_ACTIVE=1, restarted workers cannot bind their socket (as documented in lines 627-629 and the warning itself). Yet the restart still proceeds at line 692, the new process dies, and the next monitoring iteration restarts it again — an infinite loop of doomed spawns generating log noise every 10 seconds.

Consider skipping the restart (or capping per-worker retries) when network blocking is active:

Proposed fix

         if ! kill -0 "$pid" 2>/dev/null; then
             echo "[$_H] WARNING: Worker $i (PID $pid) died — restarting..."
             if [ "$NETWORK_BLOCKING_ACTIVE" = "1" ]; then
                 echo "[$_H] WARNING: Network blocking (ld.so.preload) is active. The restarted"
                 echo "[$_H]   worker may fail to bind its port because socket() is blocked for"
-                echo "[$_H]   new processes. Remaining workers continue serving requests."
+                echo "[$_H]   new processes. Skipping restart — remaining workers continue serving."
+                continue
             fi
             result=$(start_worker $i)
             WORKER_PIDS[$idx]="${result%%:*}"
             ACTUAL_WORKER_PORTS[$idx]="${result##*:}"

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	if ! kill -0 "$pid" 2>/dev/null; then
	echo "WARNING: Worker $i (PID $pid) died - restarting..."
	new_pid=$(start_worker $i)
	WORKER_PIDS[$idx]=$new_pid
	echo "[$_H] WARNING: Worker $i (PID $pid) died — restarting..."
	if [ "$NETWORK_BLOCKING_ACTIVE" = "1" ]; then
	echo "[$_H] WARNING: Network blocking (ld.so.preload) is active. The restarted"
	echo "[$_H] worker may fail to bind its port because socket() is blocked for"
	echo "[$_H] new processes. Remaining workers continue serving requests."
	fi
	result=$(start_worker $i)
	WORKER_PIDS[$idx]="${result%%:*}"
	ACTUAL_WORKER_PORTS[$idx]="${result##*:}"
	if ! kill -0 "$pid" 2>/dev/null; then
	echo "[$_H] WARNING: Worker $i (PID $pid) died — restarting..."
	if [ "$NETWORK_BLOCKING_ACTIVE" = "1" ]; then
	echo "[$_H] WARNING: Network blocking (ld.so.preload) is active. The restarted"
	echo "[$_H] worker may fail to bind its port because socket() is blocked for"
	echo "[$_H] new processes. Skipping restart — remaining workers continue serving."
	continue
	fi
	result=$(start_worker $i)
	WORKER_PIDS[$idx]="${result%%:*}"
	ACTUAL_WORKER_PORTS[$idx]="${result##*:}"

🤖 Prompt for AI Agents

In `@dockerfiles/sandbox/start-with-nginx.sh` around lines 685 - 694, When a
worker dies and NETWORK_BLOCKING_ACTIVE=1, avoid immediately restarting it to
prevent an infinite futile restart loop; modify the monitor logic around the
kill-check and restart (the block that calls start_worker and updates
WORKER_PIDS and ACTUAL_WORKER_PORTS) to either skip restarting when
NETWORK_BLOCKING_ACTIVE is set or implement a per-worker retry cap (e.g., track
restart counts in an array like WORKER_RETRIES and stop restarting after N
attempts) and log a single clear message when restarts are suppressed; ensure
you reference NETWORK_BLOCKING_ACTIVE, start_worker, WORKER_PIDS,
ACTUAL_WORKER_PORTS (and add WORKER_RETRIES if using the cap) when making the
change so the monitor stops spawning doomed processes.

[greptile-apps]

_{1 file reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Shared port file clobber

rm -f "$PORTS_REPORT_DIR/${CURRENT_NODE_SHORT}_ports.txt" runs on every node at startup. In a multi-node SLURM job, nodes can start at different times; a late-starting node can delete an already-written port report from an early node, causing the master to block until timeout in wait_for_port_reports(). Remove this unconditional delete or scope it so only the current node can clear its own file in a race-safe way (e.g., write to a temp file and mv atomically).

[greptile-apps]

NUM_WORKERS=0 breaks

This status line indexes ACTUAL_WORKER_PORTS[$((NUM_WORKERS-1))]. If NUM_WORKERS is 0 (explicitly or via nproc --all), this becomes [-1] and errors in bash. Guard earlier (validate NUM_WORKERS > 0 and exit with a clear error) before any array indexing/loops that assume at least 1 worker.

[greptile-apps]

Master detection mismatch

In SLURM mode, MASTER_NODE is taken from the expanded nodelist (e.g. node001), but the current node is detected via hostname and compared as CURRENT_NODE_SHORT == MASTER_NODE_SHORT. On many clusters hostname returns a different alias/FQDN than scontrol/nodelist uses (e.g. node001.cluster vs node001 or vice versa), which can cause multiple nodes to think they're master (or none). Prefer using SLURM-provided identity (e.g. SLURMD_NODENAME or scontrol show hostname $SLURM_NODEID output) for the local node name, and compare using the same naming scheme used to build ALL_NODES.

[greptile-apps]

Port file parsing unsafe

for endpoint in $(read_port_file ...) word-splits on whitespace. If a port file is empty/partial, read_port_file can emit blank lines and this loop will silently skip/merge tokens. Also, any unexpected whitespace will corrupt endpoints. Use a while IFS= read -r endpoint; do ...; done < <(read_port_file ...) pattern to preserve lines exactly and handle empty output deterministically.

[gwarmstrong]

@Kipok the gpt-oss test and super_49b tests pass (minus a couple unrelated RULER fluctuations)