diff --git a/.github/workflows/benchmark-tmpl.yml b/.github/workflows/benchmark-tmpl.yml index 2bcbbaeb4d..d56f7fc74f 100644 --- a/.github/workflows/benchmark-tmpl.yml +++ b/.github/workflows/benchmark-tmpl.yml @@ -214,8 +214,6 @@ env: SWEBENCH_USE_MODAL: 'true' MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }} MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }} - # These b300 nodes are currently broken. - SALLOC_EXCLUDE: 'b300-005,b300-006,b300-017' permissions: contents: read diff --git a/.github/workflows/speedbench-al.yml b/.github/workflows/speedbench-al.yml index a2adcde804..b15765ae50 100644 --- a/.github/workflows/speedbench-al.yml +++ b/.github/workflows/speedbench-al.yml @@ -75,7 +75,7 @@ permissions: env: HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }} HF_HUB_CACHE: '/mnt/hf_hub_cache/' - # Drive the single-node path in runners/launch_b300-nv.sh. MODEL is the HF id; + # Drive the single-node path in runners/launch_b300-dsxe.sh. MODEL is the HF id; # its basename (e.g. DeepSeek-V4-Pro) must be in the launcher's STAGED_MODELS so # the launcher resolves MODEL_PATH to the pre-staged local weights and mounts # them. The collector serves from MODEL_PATH (see SERVE_MODEL), so no download. diff --git a/.github/workflows/test-process-result.yml b/.github/workflows/test-process-result.yml index 04a3e701a5..51ce30e1a6 100644 --- a/.github/workflows/test-process-result.yml +++ b/.github/workflows/test-process-result.yml @@ -20,7 +20,7 @@ on: - 'runners/launch_gb300-nv.sh' - 'runners/launch_b200-nscale-slurm.sh' - 'runners/launch_b200-nscale-compat.sh' - - 'runners/launch_b300-nv.sh' + - 'runners/launch_b300-dsxe.sh' - 'runners/launch_h200-dgxc-slurm.sh' - 'runners/inject_srt_power_concurrencies.py' - 'utils/aggregate_power.py' diff --git a/benchmarks/single_node/agentic/README.md b/benchmarks/single_node/agentic/README.md index 1d23dfe2fc..3f886e605f 100644 --- a/benchmarks/single_node/agentic/README.md +++ b/benchmarks/single_node/agentic/README.md @@ -39,8 +39,8 @@ that runner label: ```yaml hardware: - cluster:b300-nv: - available-cpu-dram-mib: 2964436 + cluster:b300-dsxe: + available-cpu-dram-mib: 3977095 gpus-per-node: 8 ``` diff --git a/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm_mtp.sh index 7aa4384cf3..0b29dc021c 100755 --- a/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm_mtp.sh @@ -119,7 +119,7 @@ ROUTER_PID="" MOONCAKE_MASTER_PID="" # The generated TOTAL_CPU_DRAM_GB budget is proportional to allocated GPUs. -# On cluster:b300-nv, dram-utilization=0.95 gives both DEP4 and DEP8 356 GB per +# On cluster:b300-dsxe, dram-utilization=0.95 gives both DEP4 and DEP8 356 GB per # DP rank (1,424 GB and 2,849 GB total, respectively). TP arms remain GPU-resident. OFFLOAD_ARGS=() case "$KV_OFFLOAD_BACKEND" in diff --git a/benchmarks/single_node/speedbench/dsr1_fp4_b300_vllm.sh b/benchmarks/single_node/speedbench/dsr1_fp4_b300_vllm.sh index d0357c6b43..63b71a223b 100755 --- a/benchmarks/single_node/speedbench/dsr1_fp4_b300_vllm.sh +++ b/benchmarks/single_node/speedbench/dsr1_fp4_b300_vllm.sh @@ -28,7 +28,7 @@ # shim (the default client-side template render already enables thinking). # # Checkpoint (B300 / Blackwell): NVFP4 build nvidia/DeepSeek-R1-0528-NVFP4-v2, -# basename dsr1-fp4 on the runner (resolved by launch_b300-nv.sh). +# basename dsr1-fp4 on the runner (resolved by launch_b300-dsxe.sh). # # Usage (inside the vLLM container, on a B300 node): # export MODEL=/data/models/dsr1-fp4 diff --git a/configs/CONFIGS.md b/configs/CONFIGS.md index 7ad4418646..d30aa70aa7 100644 --- a/configs/CONFIGS.md +++ b/configs/CONFIGS.md @@ -152,13 +152,13 @@ hardware facts in the repository. It has two top-level sections: ```yaml labels: - cluster:b300-nv: - - b300-nv_01 - - b300-nv_02 + cluster:b300-dsxe: + - b300-dsxe_00 + - b300-dsxe_01 hardware: - cluster:b300-nv: - available-cpu-dram-mib: 2964436 + cluster:b300-dsxe: + available-cpu-dram-mib: 3977095 gpus-per-node: 8 ``` diff --git a/configs/runners.yaml b/configs/runners.yaml index 972d122f0b..bba672197a 100644 --- a/configs/runners.yaml +++ b/configs/runners.yaml @@ -99,6 +99,25 @@ labels: - gb200-nv_0 - gb200-nv_1 - gb200-nv_2 + b300: + - b300-dsxe_00 + - b300-dsxe_01 + - b300-dsxe_02 + - b300-dsxe_03 + - b300-dsxe_04 + - b300-dsxe_05 + - b300-dsxe_06 + - b300-dsxe_07 + - b300-dsxe_08 + - b300-dsxe_09 + - b300-dsxe_10 + - b300-dsxe_11 + - b300-dsxe_12 + - b300-dsxe_13 + - b300-dsxe_14 + - b300-dsxe_15 + - b300-dsxe_16 + - b300-dsxe_17 gb300: - gb300-nv_0 - gb300-nv_1 @@ -165,6 +184,25 @@ labels: - gb200-nv_1 - gb200-nv_2 - gb200-nv_3 + cluster:b300-dsxe: + - b300-dsxe_00 + - b300-dsxe_01 + - b300-dsxe_02 + - b300-dsxe_03 + - b300-dsxe_04 + - b300-dsxe_05 + - b300-dsxe_06 + - b300-dsxe_07 + - b300-dsxe_08 + - b300-dsxe_09 + - b300-dsxe_10 + - b300-dsxe_11 + - b300-dsxe_12 + - b300-dsxe_13 + - b300-dsxe_14 + - b300-dsxe_15 + - b300-dsxe_16 + - b300-dsxe_17 cluster:gb300-nv: - gb300-nv_0 - gb300-nv_1 @@ -221,6 +259,9 @@ hardware: cluster:b200-nscale: available-cpu-dram-mib: 2_063_920 gpus-per-node: 8 + cluster:b300-dsxe: + available-cpu-dram-mib: 3_977_095 + gpus-per-node: 8 cluster:gb200-nv: available-cpu-dram-mib: 860_160 gpus-per-node: 4 diff --git a/runners/launch_b300-dsxe.sh b/runners/launch_b300-dsxe.sh new file mode 100755 index 0000000000..c229146bfa --- /dev/null +++ b/runners/launch_b300-dsxe.sh @@ -0,0 +1,515 @@ +#!/usr/bin/bash + +# Launcher for the B300 DSXE Slurm cluster (dsxe-sa-b300-prd0), runners run as sa-gha-runner. +# +# Every cluster-specific fact lives in this block. The rest of the file is generic: +# multi-node jobs go through srt-slurm/srtctl, single-node jobs through salloc + pyxis. + +SLURM_PARTITION="batch_1" +SLURM_ACCOUNT="benchmark" + +# enroot squash images. Must be on storage every compute node mounts and writable +# by the runner user (/data/squash is root-owned, hence the per-user default). +SQUASH_DIR="/data/home/sa-gha-runner/squash" + +# Weights. MODEL_ROOT is node-local NVMe with the same layout on every compute node; +# it is read-only from the job's point of view. Anything not in STAGED_MODELS is +# downloaded into WRITABLE_MODELS_DIR (shared Lustre) by the single-node scripts. +MODEL_ROOT="/scratch/models" +WRITABLE_MODELS_DIR="/data/home/sa-gha-runner/models" + +# Official power (dcgm-power) runs use a separate, pinned producer; CI derives +# POWER_PRODUCER_SHA from the stamp this script writes. Keep in sync with the other launchers. +POWER_SRT_SLURM_URL="https://github.com/edwingao28/srt-slurm.git" +POWER_SRT_SLURM_PIN="e5c837f06a362dc888dfea2ee588e9f19c298270" + +# Directory names under MODEL_ROOT (upstream HF repo basenames). +STAGED_MODELS=( + DeepSeek-R1-0528 + DeepSeek-R1-0528-NVFP4-v2 + DeepSeek-V4-Pro + DeepSeek-V4-Pro-0813 + DeepSeek-V4-Pro-NVFP4 + GLM-5.2-NVFP4 + Kimi-K2.6-NVFP4 + Kimi-K3 + MiniMax-M3 + MiniMax-M3-MXFP8 + MiniMax-M3-NVFP4 + Qwen3.5-397B-A17B-FP8 + Qwen3.5-397B-A17B-NVFP4 + Qwen3.5-397B-A17B-NVFP4-V2 + Qwen3.8-2.4T-A95B-FP8 +) + +# srt-slurm recipes refer to models by alias (model.path in the recipe yaml). Every +# alias below is written into srtslurm.yaml, so no per-model branching is needed. +# Several aliases map to the same directory because recipes are not consistent. +declare -A MODEL_ALIASES=( + [dsr1]="DeepSeek-R1-0528-NVFP4-v2" + [dsr1-fp8]="DeepSeek-R1-0528" + [deepseek-v4-pro]="DeepSeek-V4-Pro" + [deepseek-ai/DeepSeek-V4-Pro]="DeepSeek-V4-Pro" + [glm-5.2-fp4]="GLM-5.2-NVFP4" + [nvidia/GLM-5.2-NVFP4]="GLM-5.2-NVFP4" + [kimi-k2.6-nvfp4]="Kimi-K2.6-NVFP4" + [kimi-k3]="Kimi-K3" + [kimik3]="Kimi-K3" + [moonshotai/Kimi-K3]="Kimi-K3" + [minimax-m3-nvfp4]="MiniMax-M3-NVFP4" + [nvidia/MiniMax-M3-NVFP4]="MiniMax-M3-NVFP4" + [minimax-m3-mxfp8]="MiniMax-M3-MXFP8" + [MiniMaxAI/MiniMax-M3-MXFP8]="MiniMax-M3-MXFP8" + [qwen3.5-fp4]="Qwen3.5-397B-A17B-NVFP4-V2" + [qwen3.5-fp8]="Qwen3.5-397B-A17B-FP8" + [nvidia/Qwen3.5-397B-A17B-NVFP4-V2]="Qwen3.5-397B-A17B-NVFP4-V2" +) + + +mkdir -p "$SQUASH_DIR" +set -x + +# !! KEEP THIS DEFINITION ABOVE THE IS_MULTINODE BRANCH BELOW. !! +# Both the multi-node and single-node paths call it. Bash only defines a function +# when execution reaches it, so moving this inside either branch silently removes +# it from the other and the job dies on "command not found" at import time. +# +# Import a container image into the shared squash dir. Concurrent callers target the +# same path, so serialize on a per-file lock and skip when a valid squash file exists. +# --time bounds the step; an unbounded srun hangs the job if its step is lost. +# +# The import itself must run on a compute node: enroot builds the squashfs over an +# overlay mount, which the shared filesystem cannot back, and the login host is too +# small to unpack a multi-GB image. Reading the finished file is just I/O, so probe +# it here first -- a warm cache then costs no Slurm allocation at all. The in-srun +# check under the lock stays authoritative, so a stale probe only costs one step. +import_squash_image() { + local image_ref="$1" + local sqsh="$2" + local lock="${2}.lock" + + if unsquashfs -l "$sqsh" > /dev/null 2>&1; then + echo "Squash file already present, skipping import: $sqsh" + return 0 + fi + + srun -N 1 -A "$SLURM_ACCOUNT" -p "$SLURM_PARTITION" \ + --time="${ENROOT_IMPORT_TIME_LIMIT:-120}" bash -c " + set -euo pipefail + exec 9>\"$lock\" + flock -w 3600 9 + if unsquashfs -l \"$sqsh\" > /dev/null 2>&1; then + exit 0 + fi + rm -f \"$sqsh\" + enroot import -o \"$sqsh\" \"docker://$image_ref\" + unsquashfs -l \"$sqsh\" > /dev/null + " || { echo "Error: enroot import failed for $image_ref -> $sqsh" >&2; exit 1; } + + test -r "$sqsh" || { echo "Error: squash file not readable: $sqsh" >&2; exit 1; } +} + +if [[ "$IS_MULTINODE" == "true" ]]; then + +# Validate framework +if [[ $FRAMEWORK != "dynamo-sglang" && $FRAMEWORK != "dynamo-trt" && $FRAMEWORK != "dynamo-vllm" ]]; then + echo "Unsupported framework: $FRAMEWORK. Supported frameworks are: dynamo-trt, dynamo-sglang, dynamo-vllm" + exit 1 +fi + +USES_DCGM_POWER=0 +_RECIPE_REL="${CONFIG_FILE%%:*}" +_RECIPE_SRC="$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/${_RECIPE_REL#recipes/}" +if [[ -n "$CONFIG_FILE" && -f "$_RECIPE_SRC" ]] && awk ' + /^telemetry:/ { t = 1; next } + t && /^[^ ]/ { t = 0 } + t && /^ provider: dcgm-power$/ { p = 1 } + t && /^ enabled: true$/ { e = 1 } + END { exit !(p && e) } +' "$_RECIPE_SRC"; then + USES_DCGM_POWER=1 +fi +if [[ "$USES_DCGM_POWER" == "1" && ( + "${IS_AGENTIC:-0}" == "1" || + "$MODEL_PREFIX" != "dsv4" || + "$PRECISION" != "fp4" || + ( "$FRAMEWORK" != "dynamo-sglang" && "$FRAMEWORK" != "dynamo-vllm" ) +) ]]; then + echo "Error: B300 dcgm-power is limited to fixed-sequence DSV4 FP4 dynamo-sglang/vllm" >&2 + exit 1 +fi + +# Default is the newest tag. Add a branch here to pin a ref per model / precision / +# framework when a recipe needs one, so results stay reproducible. +select_srt_slurm_version() { + if false; then + : + else + SRT_SLURM_REPO="https://github.com/NVIDIA/srt-slurm.git" + SRT_SLURM_REF="v1.0.87" + fi +} + +# --------------------------------------------------------------------------- +# srt-slurm checkout: one clone at the selected ref, plus every in-repo recipe. +# --------------------------------------------------------------------------- +SRT_REPO_DIR="srt-slurm" +rm -rf "$SRT_REPO_DIR" + +if [[ "$USES_DCGM_POWER" == "1" ]]; then + SRT_SLURM_REPO="$POWER_SRT_SLURM_URL" + SRT_SLURM_REF="$POWER_SRT_SLURM_PIN" +else + select_srt_slurm_version +fi + +echo "Cloning srt-slurm ($SRT_SLURM_REPO @ $SRT_SLURM_REF)..." +git clone "$SRT_SLURM_REPO" "$SRT_REPO_DIR" || exit 1 +cd "$SRT_REPO_DIR" || exit 1 +git checkout --quiet "$SRT_SLURM_REF" || exit 1 +git rev-parse HEAD > "$GITHUB_WORKSPACE/srt-slurm-sha.txt" +if [[ "$USES_DCGM_POWER" == "1" ]]; then + test "$(git rev-parse HEAD)" = "$POWER_SRT_SLURM_PIN" \ + || { echo "Error: srt-slurm HEAD does not match POWER_SRT_SLURM_PIN=$POWER_SRT_SLURM_PIN" >&2; exit 1; } + cp "$GITHUB_WORKSPACE/srt-slurm-sha.txt" "$GITHUB_WORKSPACE/power-producer-sha.txt" +fi + +# Recipes live in this repo; overlay all of them onto the checkout's recipes/ dir. +mkdir -p recipes +cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes" recipes || exit 1 + +if [[ "${EVAL_FRAMEWORK:-lm-eval}" != "lm-eval" ]]; then + python3 "$GITHUB_WORKSPACE/runners/patch_srt_eval_dispatch.py" "$(pwd)" || exit 1 +fi + +echo "Installing srtctl..." +export UV_INSTALL_DIR="$GITHUB_WORKSPACE/.local/bin" +curl -LsSf https://astral.sh/uv/install.sh | sh +export PATH="$UV_INSTALL_DIR:$PATH" + +uv venv "$GITHUB_WORKSPACE/.venv" +source "$GITHUB_WORKSPACE/.venv/bin/activate" +uv pip install -e . + +if ! command -v srtctl &> /dev/null; then + echo "Error: Failed to install srtctl" + exit 1 +fi + +# Map container images to local squash files +NGINX_IMAGE="nginx:1.27.4" +SQUASH_FILE="$SQUASH_DIR/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh" +NGINX_SQUASH_FILE="$SQUASH_DIR/$(echo "$NGINX_IMAGE" | sed 's/[\/:@#]/_/g').sqsh" + +# Import containers via enroot +import_squash_image "$IMAGE" "$SQUASH_FILE" +import_squash_image "$NGINX_IMAGE" "$NGINX_SQUASH_FILE" + +if [[ "$USES_DCGM_POWER" == "1" ]]; then + DCGM_EXPORTER_IMAGE="nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless" + # enroot resolves bare paths against Docker Hub; nvcr.io pulls need the registry# form + DCGM_EXPORTER_ENROOT_REF="${DCGM_EXPORTER_IMAGE/nvcr.io\//nvcr.io#}" + DCGM_EXPORTER_SQSH="$SQUASH_DIR/$(echo "$DCGM_EXPORTER_IMAGE" | sed 's/[\/:@#]/_/g').sqsh" + import_squash_image "$DCGM_EXPORTER_ENROOT_REF" "$DCGM_EXPORTER_SQSH" + sha256sum "$DCGM_EXPORTER_SQSH" > "$GITHUB_WORKSPACE/exporter-image.sha256" +fi + +export ISL="$ISL" +export OSL="$OSL" +export EVAL_ONLY="${EVAL_ONLY:-false}" + +# --------------------------------------------------------------------------- +# srtslurm.yaml: cluster defaults, every model alias, container aliases. +# --------------------------------------------------------------------------- +SRTCTL_ROOT="${GITHUB_WORKSPACE}/${SRT_REPO_DIR}" +echo "Creating srtslurm.yaml configuration..." +{ + cat < srtslurm.yaml + +echo "Generated srtslurm.yaml:" +cat srtslurm.yaml + +echo "Running make setup..." +make setup ARCH=x86_64 + +# Export eval-related env vars for srt-slurm post-benchmark eval +export INFMAX_WORKSPACE="$GITHUB_WORKSPACE" + +echo "Submitting job with srtctl..." + +if [[ -z "$CONFIG_FILE" ]]; then + echo "Error: CONFIG_FILE is not set. The srt-slurm path requires a CONFIG_FILE in additional-settings." >&2 + echo "Config: MODEL_PREFIX=${MODEL_PREFIX} PRECISION=${PRECISION} FRAMEWORK=${FRAMEWORK}" >&2 + exit 1 +fi + +# Resolve the recipe path before editing it. CONFIG_FILE may include an +# srt-slurm matrix selector such as :zip_override_dep4_dep8[0]. +CONFIG_PATH="${CONFIG_FILE%%:*}" +if [[ ! -f "$CONFIG_PATH" ]]; then + echo "Error: CONFIG_FILE does not exist after srt-slurm setup: $CONFIG_PATH" >&2 + exit 1 +fi + +# Override the job name in the recipe with the runner name. +sed -i "s/^name:.*/name: \"${RUNNER_NAME}\"/" "$CONFIG_PATH" +if [[ "${EVAL_ONLY:-false}" == "true" ]]; then + python3 "$GITHUB_WORKSPACE/runners/inject_synthetic_acceptance.py" \ + "$CONFIG_PATH" "$FRAMEWORK" || exit 1 +fi + +# Weights live on node-local MODEL_ROOT, which this login host cannot stat, so +# srtctl's preflight model.path check is always skipped. Runtime loading still +# validates the path on the compute nodes. +SRTCTL_APPLY_ARGS=( + -f "$CONFIG_FILE" + --no-preflight + --tags "b300,${MODEL_PREFIX},${PRECISION},${ISL}x${OSL},infmax-$(date +%Y%m%d)" +) +SRTCTL_OUTPUT=$(srtctl apply "${SRTCTL_APPLY_ARGS[@]}" 2>&1) +echo "$SRTCTL_OUTPUT" + +# Extract JOB_ID from srtctl output +JOB_ID=$(echo "$SRTCTL_OUTPUT" | grep -oP '✅ Job \K[0-9]+' || echo "$SRTCTL_OUTPUT" | grep -oP 'Job \K[0-9]+') + +set +x + +if [ -z "$JOB_ID" ]; then + echo "Error: Failed to extract JOB_ID from srtctl output" + exit 1 +fi + +echo "Extracted JOB_ID: $JOB_ID" + +# Use the JOB_ID to find the logs directory +# srtctl creates logs in outputs/JOB_ID/logs/ +LOGS_DIR="outputs/$JOB_ID/logs" +LOG_FILE="$LOGS_DIR/sweep_${JOB_ID}.log" + +# Wait for log file to appear (also check job is still alive) +while ! ls "$LOG_FILE" &>/dev/null; do + if ! squeue -j "$JOB_ID" --noheader 2>/dev/null | grep -q "$JOB_ID"; then + echo "ERROR: Job $JOB_ID failed before creating log file" + scontrol show job "$JOB_ID" + exit 1 + fi + echo "Waiting for JOB_ID $JOB_ID to begin and $LOG_FILE to appear..." + sleep 5 +done + +# Poll for job completion in background +( + while squeue -j "$JOB_ID" --noheader 2>/dev/null | grep -q "$JOB_ID"; do + sleep 10 + done +) & +POLL_PID=$! + +echo "Tailing LOG_FILE: $LOG_FILE" + +# Stream the log file until job completes (-F follows by name, polls instead of inotify for NFS) +tail -F -s 2 -n+1 "$LOG_FILE" --pid=$POLL_PID 2>/dev/null + +wait $POLL_PID + +set -x + +echo "Job $JOB_ID completed!" +echo "Collecting results..." + +if [ ! -d "$LOGS_DIR" ]; then + echo "Warning: Logs directory not found at $LOGS_DIR" + exit 1 +fi + +echo "Found logs directory: $LOGS_DIR" + +if [[ "$USES_DCGM_POWER" == "1" ]]; then + mkdir -p "$LOGS_DIR/power" + cp "$GITHUB_WORKSPACE/exporter-image.sha256" "$LOGS_DIR/power/exporter-image.sha256" + cp "$GITHUB_WORKSPACE/power-producer-sha.txt" "$LOGS_DIR/power/power-producer-sha.txt" +fi + +cp -r "$LOGS_DIR" "$GITHUB_WORKSPACE/LOGS" +tar czf "$GITHUB_WORKSPACE/multinode_server_logs.tar.gz" -C "$LOGS_DIR" . + +if [[ "${EVAL_ONLY:-false}" != "true" ]]; then + # Find all result subdirectories + RESULT_SUBDIRS=$(find "$LOGS_DIR" -maxdepth 1 -type d -name "*isl*osl*" 2>/dev/null) + + if [ -z "$RESULT_SUBDIRS" ]; then + echo "Warning: No result subdirectories found in $LOGS_DIR" + else + # Process results from all configurations + for result_subdir in $RESULT_SUBDIRS; do + echo "Processing result subdirectory: $result_subdir" + + # Extract configuration info from directory name + CONFIG_NAME=$(basename "$result_subdir") + + # Find all result JSON files + RESULT_FILES=$(find "$result_subdir" -name "results_concurrency_*.json" 2>/dev/null) + + for result_file in $RESULT_FILES; do + if [ -f "$result_file" ]; then + # Extract metadata from filename + # Files may be "results_concurrency_N_gpus_G_ctx_C_gen_D.json" (disagg) or "results_concurrency_N_gpus_G.json" (non-disagg) + filename=$(basename "$result_file") + concurrency=$(echo "$filename" | sed -n 's/results_concurrency_\([0-9]*\)_gpus_.*/\1/p') + gpus=$(echo "$filename" | sed -n 's/results_concurrency_[0-9]*_gpus_\([0-9][0-9]*\).*/\1/p') + ctx=$(echo "$filename" | sed -n 's/.*_ctx_\([0-9]*\)_gen_.*/\1/p') + gen=$(echo "$filename" | sed -n 's/.*_gen_\([0-9]*\)\.json/\1/p') + + echo "Processing concurrency $concurrency with $gpus GPUs (ctx: $ctx, gen: $gen): $result_file" + + if [ -n "$ctx" ] && [ -n "$gen" ]; then + WORKSPACE_RESULT_FILE="$GITHUB_WORKSPACE/${RESULT_FILENAME}_${CONFIG_NAME}_conc${concurrency}_gpus_${gpus}_ctx_${ctx}_gen_${gen}.json" + else + WORKSPACE_RESULT_FILE="$GITHUB_WORKSPACE/${RESULT_FILENAME}_${CONFIG_NAME}_conc${concurrency}_gpus_${gpus}.json" + fi + cp "$result_file" "$WORKSPACE_RESULT_FILE" + + echo "Copied result file to: $WORKSPACE_RESULT_FILE" + fi + done + done + fi + + echo "All result files processed" +else + echo "EVAL_ONLY=true: Skipping benchmark result collection" +fi + +# Collect eval results if eval was requested +if [[ "${RUN_EVAL:-false}" == "true" || "${EVAL_ONLY:-false}" == "true" ]]; then + EVAL_DIR="$LOGS_DIR/eval_results" + if [ -d "$EVAL_DIR" ]; then + echo "Extracting eval results from $EVAL_DIR" + shopt -s nullglob + for eval_file in "$EVAL_DIR"/*; do + [ -f "$eval_file" ] || continue + cp "$eval_file" "$GITHUB_WORKSPACE/" + echo "Copied eval artifact: $(basename "$eval_file")" + done + shopt -u nullglob + else + echo "WARNING: RUN_EVAL=true but no eval results found at $EVAL_DIR" + fi +fi + +# Clean up srt-slurm outputs to prevent NFS silly-rename lock files +# from blocking the next job's checkout on this runner +echo "Cleaning up srt-slurm outputs..." +for i in 1 2 3 4 5; do + rm -rf outputs 2>/dev/null && break + echo "Retry $i/5: Waiting for NFS locks to release..." + sleep 10 +done +find . -name '.nfs*' -delete 2>/dev/null || true + +else + # HF_HUB_CACHE is set to help with dataset download inside the container + # for eval jobs. + export HF_HUB_CACHE="$HOME/.cache/huggingface" + + # MODEL stays the HF id for the client; MODEL_PATH is where the server reads + # weights. Only the root holding MODEL_PATH is mounted -- mounting both roots + # makes pyxis fail whenever the unused one is absent on the node. + MODEL_BASENAME="${MODEL##*/}" + if [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then + MODEL_MOUNT_DIR="$MODEL_ROOT" + else + MODEL_MOUNT_DIR="$WRITABLE_MODELS_DIR" + mkdir -p "$WRITABLE_MODELS_DIR" + fi + export MODEL_PATH="${MODEL_MOUNT_DIR}/${MODEL_BASENAME}" + + SQUASH_FILE="$SQUASH_DIR/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh" + SPEC_SUFFIX=$([[ "$SPEC_DECODING" == "mtp" ]] && printf '_mtp' || printf '') + # Prefer a framework-tagged script (e.g. dsv4_fp4_b300_sglang.sh); fall back to + # the untagged historical name for scripts that haven't been retagged yet. + BENCH_BASE="benchmarks/single_node/${SCENARIO_SUBDIR}${EXP_NAME%%_*}_${PRECISION}_b300" + BENCH_SCRIPT="${BENCH_BASE}_${FRAMEWORK}${SPEC_SUFFIX}.sh" + if [[ ! -f "$BENCH_SCRIPT" ]]; then + LEGACY_FW_SUFFIX=$([[ "$FRAMEWORK" == "trt" ]] && printf '_trt' || printf '') + BENCH_SCRIPT="${BENCH_BASE}${LEGACY_FW_SUFFIX}${SPEC_SUFFIX}.sh" + fi + + # Allow callers (e.g. the speedbench-al.yml AL-collection workflow) to run a + # specific script instead of the auto-selected throughput benchmark. + if [[ -n "${BENCH_SCRIPT_OVERRIDE:-}" ]]; then + BENCH_SCRIPT="$BENCH_SCRIPT_OVERRIDE" + fi + + # These images install sglang editable under /workspace, so the default + # workspace bind-mount masks the install and breaks `import sglang`. Mount at + # /ix instead; drop this once the images stop installing there. + if [[ "$IMAGE" == *deepseek-v4-blackwell* || "$IMAGE" == *deepseek-v4-bw-ultra* || "$IMAGE" == *deepseek-v4-b300* || "$IMAGE" == *sglang-b300* ]]; then + CONTAINER_MOUNT_DIR=/ix + else + CONTAINER_MOUNT_DIR=/workspace + fi + + import_squash_image "$IMAGE" "$SQUASH_FILE" + + export GPU_COUNT="${GPU_COUNT:-${TP:?TP must be set}}" + + SALLOC_ARGS=( + --partition="$SLURM_PARTITION" + --account="$SLURM_ACCOUNT" + -N 1 + --gres="gpu:$GPU_COUNT" + --exclusive + --mem=0 + --time="${SALLOC_TIME_LIMIT:-480}" + --no-shell + --job-name="$RUNNER_NAME" + ) + # Optional escape hatch for taking a bad node out of rotation without a code change. + if [[ -n "${SALLOC_EXCLUDE:-}" ]]; then + SALLOC_ARGS+=(--exclude="$SALLOC_EXCLUDE") + fi + salloc "${SALLOC_ARGS[@]}" + JOB_ID=$(squeue --name="$RUNNER_NAME" -u "$USER" -h -o %A | head -n1) + + CONTAINER_MOUNTS=( + "$GITHUB_WORKSPACE:$CONTAINER_MOUNT_DIR" + "$MODEL_MOUNT_DIR:$MODEL_MOUNT_DIR" + ) + CONTAINER_MOUNTS_ARG=$(IFS=,; printf '%s' "${CONTAINER_MOUNTS[*]}") + + srun --jobid="$JOB_ID" \ + --mpi=none \ + --container-image="$SQUASH_FILE" \ + --container-mounts="$CONTAINER_MOUNTS_ARG" \ + --no-container-mount-home \ + --container-remap-root \ + --container-workdir="$CONTAINER_MOUNT_DIR" \ + --no-container-entrypoint --export=ALL,PORT=8888 \ + bash "$BENCH_SCRIPT" + +fi diff --git a/runners/launch_b300-nv.sh b/runners/launch_b300-nv.sh deleted file mode 100644 index b56c03a56b..0000000000 --- a/runners/launch_b300-nv.sh +++ /dev/null @@ -1,590 +0,0 @@ -#!/usr/bin/bash - -# System-specific configuration for B300 NV Slurm cluster (sa-shared) -SLURM_PARTITION="batch_1" -SLURM_ACCOUNT="benchmark" -POWER_SRT_SLURM_URL="https://github.com/edwingao28/srt-slurm.git" -POWER_SRT_SLURM_PIN="e5c837f06a362dc888dfea2ee588e9f19c298270" -# b300-018 repeatedly times out UCX/NIXL transfers; allow an empty override to disable this. -MINIMAX_M3_SLURM_EXCLUDED_NODELIST="${MINIMAX_M3_SLURM_EXCLUDED_NODELIST-b300-018}" - -set -x - -if [[ "$IS_MULTINODE" == "true" ]]; then - -# Validate framework -if [[ $FRAMEWORK != "dynamo-sglang" && $FRAMEWORK != "dynamo-trt" && $FRAMEWORK != "dynamo-vllm" ]]; then - echo "Unsupported framework: $FRAMEWORK. Supported frameworks are: dynamo-trt, dynamo-sglang, dynamo-vllm" - exit 1 -fi - -USES_DCGM_POWER=0 -_RECIPE_REL="${CONFIG_FILE%%:*}" -_RECIPE_SRC="$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/${_RECIPE_REL#recipes/}" -if [[ -n "$CONFIG_FILE" && -f "$_RECIPE_SRC" ]] && awk ' - /^telemetry:/ { t = 1; next } - t && /^[^ ]/ { t = 0 } - t && /^ provider: dcgm-power$/ { p = 1 } - t && /^ enabled: true$/ { e = 1 } - END { exit !(p && e) } -' "$_RECIPE_SRC"; then - USES_DCGM_POWER=1 -fi -if [[ "$USES_DCGM_POWER" == "1" && ( - "${IS_AGENTIC:-0}" == "1" || - "$MODEL_PREFIX" != "dsv4" || - "$PRECISION" != "fp4" || - ( "$FRAMEWORK" != "dynamo-sglang" && "$FRAMEWORK" != "dynamo-vllm" ) -) ]]; then - echo "Error: B300 dcgm-power is limited to fixed-sequence DSV4 FP4 dynamo-sglang/vllm" >&2 - exit 1 -fi - -# MODEL_PATH: Override with pre-downloaded paths on B300 runner -# The yaml files specify HuggingFace model IDs for portability, but we use -# local paths to avoid repeated downloading on the shared B300 cluster. -if [[ $MODEL_PREFIX == "dsr1" && $PRECISION == "fp4" ]]; then - export MODEL_PATH="/data/models/dsr1-fp4" - export SERVED_MODEL_NAME="deepseek-r1-fp4" - export SRT_SLURM_MODEL_PREFIX="dsr1" -elif [[ $MODEL_PREFIX == "dsr1" && $PRECISION == "fp8" ]]; then - export MODEL_PATH="/data/models/dsr1-fp8" - export SERVED_MODEL_NAME="deepseek-r1-fp8" - export SRT_SLURM_MODEL_PREFIX="dsr1-fp8" -elif [[ $MODEL_PREFIX == "dsv4" && $PRECISION == "fp4" && $FRAMEWORK == "dynamo-vllm" ]]; then - SELECTED_MODEL_PATH="" - if [[ -n "${MODEL_PATH:-}" && -d "${MODEL_PATH}" ]]; then - SELECTED_MODEL_PATH="$MODEL_PATH" - else - for candidate in /data/models/dsv4-pro /data/models/deepseek-v4-pro /data/models/DeepSeek-V4-Pro; do - if [[ -d "$candidate" ]]; then - SELECTED_MODEL_PATH="$candidate" - break - fi - done - fi - export MODEL_PATH="${SELECTED_MODEL_PATH:-/data/models/dsv4-pro}" - export SRT_SLURM_MODEL_PREFIX="deepseek-v4-pro" -elif [[ $MODEL_PREFIX == "dsv4" && $PRECISION == "fp4" && $FRAMEWORK == "dynamo-sglang" ]]; then - export MODEL_PATH="${MODEL_PATH:-/scratch/models/DeepSeek-V4-Pro}" - export SRT_SLURM_MODEL_PREFIX="deepseek-v4-pro" -elif [[ $MODEL_PREFIX == "minimaxm2.5" && $PRECISION == "fp4" && $FRAMEWORK == "dynamo-vllm" ]]; then - export MODEL_PATH="/data/models/MiniMax-M2.5-NVFP4" - export SRT_SLURM_MODEL_PREFIX="minimax-m2.5-nvfp4" -elif [[ $MODEL_PREFIX == "minimaxm2.5" && $PRECISION == "fp8" && $FRAMEWORK == "dynamo-vllm" ]]; then - export MODEL_PATH="/data/models/MiniMax-M2.5" - export SRT_SLURM_MODEL_PREFIX="minimax-m2.5-fp8" -elif [[ $MODEL_PREFIX == "minimaxm3" && $PRECISION == "fp4" && $FRAMEWORK == "dynamo-vllm" ]]; then - export MODEL_PATH="/scratch/models/MiniMax-M3-NVFP4" - export SRT_SLURM_MODEL_PREFIX="nvidia/MiniMax-M3-NVFP4" -elif [[ $MODEL_PREFIX == "minimaxm3" && $PRECISION == "fp8" && $FRAMEWORK == "dynamo-vllm" ]]; then - export MODEL_PATH="/data/models/MiniMax-M3-MXFP8" - export SRT_SLURM_MODEL_PREFIX="MiniMaxAI/MiniMax-M3-MXFP8" -else - echo "Unsupported model: $MODEL_PREFIX-$PRECISION. Supported models are: dsr1-fp4, dsr1-fp8, dsv4-fp4 with dynamo-vllm or dynamo-sglang, minimaxm2.5-fp4 with dynamo-vllm, minimaxm2.5-fp8 with dynamo-vllm, minimaxm3-fp4 with dynamo-vllm, minimaxm3-fp8 with dynamo-vllm" - exit 1 -fi - -echo "Cloning srt-slurm repository..." -SRT_REPO_DIR="srt-slurm" -SRTCTL_SETUP_SCRIPT="" -if [ -d "$SRT_REPO_DIR" ]; then - echo "Removing existing $SRT_REPO_DIR..." - rm -rf "$SRT_REPO_DIR" -fi - -# TODO(CJQ): make first class upon srt-slurm upstream refactor -if [[ "$USES_DCGM_POWER" == "1" ]]; then - git clone "$POWER_SRT_SLURM_URL" "$SRT_REPO_DIR" || exit 1 - cd "$SRT_REPO_DIR" || exit 1 - git checkout "$POWER_SRT_SLURM_PIN" || exit 1 - test "$(git rev-parse HEAD)" = "$POWER_SRT_SLURM_PIN" || { echo "Error: srt-slurm HEAD does not match POWER_SRT_SLURM_PIN=$POWER_SRT_SLURM_PIN" >&2; exit 1; } - git rev-parse HEAD > "$GITHUB_WORKSPACE/power-producer-sha.txt" - if [[ "$FRAMEWORK" == "dynamo-sglang" ]]; then - mkdir -p recipes/sglang/deepseek-v4 - cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4" recipes/sglang/deepseek-v4 - else - mkdir -p recipes/vllm/deepseek-v4 - cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4" recipes/vllm/deepseek-v4 - fi -elif [[ "$IS_AGENTIC" == "1" ]]; then - git clone --branch cam/sa-submission-q2-2026 --single-branch https://github.com/cquil11/srt-slurm-nv.git "$SRT_REPO_DIR" - cd "$SRT_REPO_DIR" || exit 1 -elif [[ $FRAMEWORK == "dynamo-vllm" && $MODEL_PREFIX == "dsv4" ]]; then - git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" - cd "$SRT_REPO_DIR" || exit 1 - git checkout aflowers/vllm-gb200-v0.20.0 - mkdir -p recipes/vllm/deepseek-v4 - cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4" recipes/vllm/deepseek-v4 -elif [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "dsv4" && $PRECISION == "fp4" ]]; then - git clone --branch main --single-branch https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" - cd "$SRT_REPO_DIR" || exit 1 - git checkout c180328b98c3793ca84a1e24a030f90545eb7d5d || exit 1 - mkdir -p recipes/sglang/deepseek-v4 - cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4" recipes/sglang/deepseek-v4 -elif [[ $FRAMEWORK == "dynamo-vllm" && $MODEL_PREFIX == "minimaxm3" && $PRECISION == "fp4" && "$CONFIG_FILE" == recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/*.yaml ]]; then - git clone --branch main --single-branch https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" - cd "$SRT_REPO_DIR" || exit 1 - git checkout c1b6b5c97f323baefad577d70c4e8392b6f537d9 - mkdir -p recipes/vllm/minimax-m3 - cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3" recipes/vllm/minimax-m3 -elif [[ $FRAMEWORK == "dynamo-vllm" && $MODEL_PREFIX == "minimaxm3" && $PRECISION == "fp4" && "$CONFIG_FILE" == recipes/vllm/minimax-m3/b300-fp4/8k1k/*-tp1-*.yaml ]]; then - git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" - cd "$SRT_REPO_DIR" || exit 1 - git checkout c1fb6989fc5aca803b4ca0f2d17d8be85fad9732 - mkdir -p recipes/vllm/minimax-m3 - cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3" recipes/vllm/minimax-m3 -elif [[ $FRAMEWORK == "dynamo-vllm" && $MODEL_PREFIX == "minimaxm3" && ( $PRECISION == "fp4" || $PRECISION == "fp8" ) ]]; then - git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" - cd "$SRT_REPO_DIR" || exit 1 - git checkout sa-submission-q2-2026 - mkdir -p recipes/vllm/minimax-m3 - cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3" recipes/vllm/minimax-m3 - # NVIDIA/srt-slurm#38 - git show 22d46ba9971615016d2339c9ffbc7b4597accfad --format= -- src/srtctl/core/ip_utils/get_node_ip.sh | git apply - || exit 1 - if [[ -n "$SRTCTL_SETUP_SCRIPT" ]]; then - cp \ - "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/configs/$SRTCTL_SETUP_SCRIPT" \ - "configs/$SRTCTL_SETUP_SCRIPT" - fi -else - git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" - cd "$SRT_REPO_DIR" || exit 1 - git checkout sa-submission-q2-2026 -fi -if [[ "${EVAL_FRAMEWORK:-lm-eval}" != "lm-eval" ]]; then - python3 "$GITHUB_WORKSPACE/runners/patch_srt_eval_dispatch.py" "$(pwd)" \ - || exit 1 -fi - - -echo "Installing srtctl..." -export UV_INSTALL_DIR="$GITHUB_WORKSPACE/.local/bin" -curl -LsSf https://astral.sh/uv/install.sh | sh -export PATH="$UV_INSTALL_DIR:$PATH" - -uv venv "$GITHUB_WORKSPACE/.venv" -source "$GITHUB_WORKSPACE/.venv/bin/activate" -uv pip install -e . - -if ! command -v srtctl &> /dev/null; then - echo "Error: Failed to install srtctl" - exit 1 -fi - -# Map container images to local squash files -NGINX_IMAGE="nginx:1.27.4" -SQUASH_FILE="/data/squash/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh" -NGINX_SQUASH_FILE="/data/squash/$(echo "$NGINX_IMAGE" | sed 's/[\/:@#]/_/g').sqsh" - -# Import containers via enroot -srun -N 1 -A $SLURM_ACCOUNT -p $SLURM_PARTITION bash -c "enroot import -o $SQUASH_FILE docker://$IMAGE" -srun -N 1 -A $SLURM_ACCOUNT -p $SLURM_PARTITION bash -c "enroot import -o $NGINX_SQUASH_FILE docker://$NGINX_IMAGE" - -if [[ "$USES_DCGM_POWER" == "1" ]]; then - DCGM_EXPORTER_IMAGE="nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless" - # enroot resolves bare paths against Docker Hub; nvcr.io pulls need the registry# form - DCGM_EXPORTER_ENROOT_REF="${DCGM_EXPORTER_IMAGE/nvcr.io\//nvcr.io#}" - DCGM_EXPORTER_SQSH="/data/squash/$(echo "$DCGM_EXPORTER_IMAGE" | sed 's/[\/:@#]/_/g').sqsh" - DCGM_EXPORTER_LOCK="${DCGM_EXPORTER_SQSH}.lock" - srun -N 1 -A "$SLURM_ACCOUNT" -p "$SLURM_PARTITION" bash -c " - set -euo pipefail - exec 9>\"$DCGM_EXPORTER_LOCK\" - flock -w 1800 9 - if unsquashfs -l \"$DCGM_EXPORTER_SQSH\" > /dev/null 2>&1; then - exit 0 - fi - rm -f \"$DCGM_EXPORTER_SQSH\" - enroot import -o \"$DCGM_EXPORTER_SQSH\" \"docker://$DCGM_EXPORTER_ENROOT_REF\" - unsquashfs -l \"$DCGM_EXPORTER_SQSH\" > /dev/null - " || exit 1 - test -r "$DCGM_EXPORTER_SQSH" || { echo "Error: DCGM exporter squash not readable: $DCGM_EXPORTER_SQSH" >&2; exit 1; } - sha256sum "$DCGM_EXPORTER_SQSH" > "$GITHUB_WORKSPACE/exporter-image.sha256" -fi - -export ISL="$ISL" -export OSL="$OSL" -export EVAL_ONLY="${EVAL_ONLY:-false}" - -# Create srtslurm.yaml for srtctl -SRTCTL_ROOT="${GITHUB_WORKSPACE}/${SRT_REPO_DIR}" -echo "Creating srtslurm.yaml configuration..." -cat > srtslurm.yaml <&2; exit 1; } -fi - -echo "Generated srtslurm.yaml:" -cat srtslurm.yaml - -echo "Running make setup..." -make setup ARCH=x86_64 - -# Export eval-related env vars for srt-slurm post-benchmark eval -export INFMAX_WORKSPACE="$GITHUB_WORKSPACE" - -echo "Submitting job with srtctl..." - -if [[ -z "$CONFIG_FILE" ]]; then - echo "Error: CONFIG_FILE is not set. The srt-slurm path requires a CONFIG_FILE in additional-settings." >&2 - echo "Config: MODEL_PREFIX=${MODEL_PREFIX} PRECISION=${PRECISION} FRAMEWORK=${FRAMEWORK}" >&2 - exit 1 -fi - -# Resolve the recipe path before editing it. CONFIG_FILE may include an -# srt-slurm matrix selector such as :zip_override_dep4_dep8[0]. -CONFIG_PATH="${CONFIG_FILE%%:*}" -if [[ ! -f "$CONFIG_PATH" ]]; then - echo "Error: CONFIG_FILE does not exist after srt-slurm setup: $CONFIG_PATH" >&2 - exit 1 -fi - -# Override the job name in the recipe with the runner name. -sed -i "s/^name:.*/name: \"${RUNNER_NAME}\"/" "$CONFIG_PATH" -if [[ "$MODEL_PREFIX" == "minimaxm3" && -n "$MINIMAX_M3_SLURM_EXCLUDED_NODELIST" ]]; then - sed -i "/^name:.*/a sbatch_directives:\n exclude: \"${MINIMAX_M3_SLURM_EXCLUDED_NODELIST}\"" "$CONFIG_PATH" -fi -if [[ "${EVAL_ONLY:-false}" == "true" ]]; then - python3 "$GITHUB_WORKSPACE/runners/inject_synthetic_acceptance.py" \ - "$CONFIG_PATH" "$FRAMEWORK" || exit 1 -fi -SRTCTL_APPLY_ARGS=( - -f "$CONFIG_FILE" - --tags "b300,${MODEL_PREFIX},${PRECISION},${ISL}x${OSL},infmax-$(date +%Y%m%d)" -) -# The MTP and TP1 8k1k recipes use newer srt-slurm revisions whose preflight checks -# model.path on this GHA login host. MiniMax-M3 NVFP4 is intentionally staged -# under compute-node-local /scratch (as in the original B300 submission), so -# the login host cannot stat it even though workers can. Keep this bypass -# scoped to those recipe sets; runtime model loading still validates the path. -if [[ $FRAMEWORK == "dynamo-vllm" && $MODEL_PREFIX == "minimaxm3" && $PRECISION == "fp4" && ( "$CONFIG_FILE" == recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/*.yaml || "$CONFIG_FILE" == recipes/vllm/minimax-m3/b300-fp4/8k1k/*-tp1-*.yaml ) ]]; then - SRTCTL_APPLY_ARGS+=(--no-preflight) -fi -if [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "dsv4" && "$MODEL_PATH" == /scratch/models/* ]]; then - SRTCTL_APPLY_ARGS+=(--no-preflight) -fi -if [[ -n "$SRTCTL_SETUP_SCRIPT" ]]; then - SRTCTL_APPLY_ARGS+=(--setup-script "$SRTCTL_SETUP_SCRIPT") -fi -SRTCTL_OUTPUT=$(srtctl apply "${SRTCTL_APPLY_ARGS[@]}" 2>&1) -echo "$SRTCTL_OUTPUT" - -# Extract JOB_ID from srtctl output -JOB_ID=$(echo "$SRTCTL_OUTPUT" | grep -oP '✅ Job \K[0-9]+' || echo "$SRTCTL_OUTPUT" | grep -oP 'Job \K[0-9]+') - -set +x - -if [ -z "$JOB_ID" ]; then - echo "Error: Failed to extract JOB_ID from srtctl output" - exit 1 -fi - -if [[ "$MODEL_PREFIX" == "minimaxm3" && -n "$MINIMAX_M3_SLURM_EXCLUDED_NODELIST" ]]; then - SBATCH_SCRIPT="outputs/$JOB_ID/sbatch_script.sh" - if ! grep -Fq "#SBATCH --exclude=${MINIMAX_M3_SLURM_EXCLUDED_NODELIST}" "$SBATCH_SCRIPT"; then - echo "Error: Slurm node exclusion was not rendered in $SBATCH_SCRIPT" >&2 - scancel "$JOB_ID" || true - exit 1 - fi -fi - -echo "Extracted JOB_ID: $JOB_ID" - -# Use the JOB_ID to find the logs directory -# srtctl creates logs in outputs/JOB_ID/logs/ -LOGS_DIR="outputs/$JOB_ID/logs" -LOG_FILE="$LOGS_DIR/sweep_${JOB_ID}.log" - -# Wait for log file to appear (also check job is still alive) -while ! ls "$LOG_FILE" &>/dev/null; do - if ! squeue -j "$JOB_ID" --noheader 2>/dev/null | grep -q "$JOB_ID"; then - echo "ERROR: Job $JOB_ID failed before creating log file" - scontrol show job "$JOB_ID" - exit 1 - fi - echo "Waiting for JOB_ID $JOB_ID to begin and $LOG_FILE to appear..." - sleep 5 -done - -# Poll for job completion in background -( - while squeue -j "$JOB_ID" --noheader 2>/dev/null | grep -q "$JOB_ID"; do - sleep 10 - done -) & -POLL_PID=$! - -echo "Tailing LOG_FILE: $LOG_FILE" - -# Stream the log file until job completes (-F follows by name, polls instead of inotify for NFS) -tail -F -s 2 -n+1 "$LOG_FILE" --pid=$POLL_PID 2>/dev/null - -wait $POLL_PID - -set -x - -echo "Job $JOB_ID completed!" -echo "Collecting results..." - -if [ ! -d "$LOGS_DIR" ]; then - echo "Warning: Logs directory not found at $LOGS_DIR" - exit 1 -fi - -echo "Found logs directory: $LOGS_DIR" - -if [[ "$USES_DCGM_POWER" == "1" ]]; then - mkdir -p "$LOGS_DIR/power" - cp "$GITHUB_WORKSPACE/exporter-image.sha256" "$LOGS_DIR/power/exporter-image.sha256" - cp "$GITHUB_WORKSPACE/power-producer-sha.txt" "$LOGS_DIR/power/power-producer-sha.txt" -fi - -cp -r "$LOGS_DIR" "$GITHUB_WORKSPACE/LOGS" -tar czf "$GITHUB_WORKSPACE/multinode_server_logs.tar.gz" -C "$LOGS_DIR" . - -if [[ "${EVAL_ONLY:-false}" != "true" ]]; then - # Find all result subdirectories - RESULT_SUBDIRS=$(find "$LOGS_DIR" -maxdepth 1 -type d -name "*isl*osl*" 2>/dev/null) - - if [ -z "$RESULT_SUBDIRS" ]; then - echo "Warning: No result subdirectories found in $LOGS_DIR" - else - # Process results from all configurations - for result_subdir in $RESULT_SUBDIRS; do - echo "Processing result subdirectory: $result_subdir" - - # Extract configuration info from directory name - CONFIG_NAME=$(basename "$result_subdir") - - # Find all result JSON files - RESULT_FILES=$(find "$result_subdir" -name "results_concurrency_*.json" 2>/dev/null) - - for result_file in $RESULT_FILES; do - if [ -f "$result_file" ]; then - # Extract metadata from filename - # Files may be "results_concurrency_N_gpus_G_ctx_C_gen_D.json" (disagg) or "results_concurrency_N_gpus_G.json" (non-disagg) - filename=$(basename "$result_file") - concurrency=$(echo "$filename" | sed -n 's/results_concurrency_\([0-9]*\)_gpus_.*/\1/p') - gpus=$(echo "$filename" | sed -n 's/results_concurrency_[0-9]*_gpus_\([0-9][0-9]*\).*/\1/p') - ctx=$(echo "$filename" | sed -n 's/.*_ctx_\([0-9]*\)_gen_.*/\1/p') - gen=$(echo "$filename" | sed -n 's/.*_gen_\([0-9]*\)\.json/\1/p') - - echo "Processing concurrency $concurrency with $gpus GPUs (ctx: $ctx, gen: $gen): $result_file" - - if [ -n "$ctx" ] && [ -n "$gen" ]; then - WORKSPACE_RESULT_FILE="$GITHUB_WORKSPACE/${RESULT_FILENAME}_${CONFIG_NAME}_conc${concurrency}_gpus_${gpus}_ctx_${ctx}_gen_${gen}.json" - else - WORKSPACE_RESULT_FILE="$GITHUB_WORKSPACE/${RESULT_FILENAME}_${CONFIG_NAME}_conc${concurrency}_gpus_${gpus}.json" - fi - cp "$result_file" "$WORKSPACE_RESULT_FILE" - - echo "Copied result file to: $WORKSPACE_RESULT_FILE" - fi - done - done - fi - - echo "All result files processed" -else - echo "EVAL_ONLY=true: Skipping benchmark result collection" -fi - -# Collect eval results if eval was requested -if [[ "${RUN_EVAL:-false}" == "true" || "${EVAL_ONLY:-false}" == "true" ]]; then - EVAL_DIR="$LOGS_DIR/eval_results" - if [ -d "$EVAL_DIR" ]; then - echo "Extracting eval results from $EVAL_DIR" - shopt -s nullglob - for eval_file in "$EVAL_DIR"/*; do - [ -f "$eval_file" ] || continue - cp "$eval_file" "$GITHUB_WORKSPACE/" - echo "Copied eval artifact: $(basename "$eval_file")" - done - shopt -u nullglob - else - echo "WARNING: RUN_EVAL=true but no eval results found at $EVAL_DIR" - fi -fi - -# Clean up srt-slurm outputs to prevent NFS silly-rename lock files -# from blocking the next job's checkout on this runner -echo "Cleaning up srt-slurm outputs..." -for i in 1 2 3 4 5; do - rm -rf outputs 2>/dev/null && break - echo "Retry $i/5: Waiting for NFS locks to release..." - sleep 10 -done -find . -name '.nfs*' -delete 2>/dev/null || true - -else - # HF_HUB_CACHE is set to help with dataset download inside the container - # for eval jobs. Can be updated to some other path on the cluster and - # mounted just like HF_HUB_CACHE_MOUNT. - export HF_HUB_CACHE="$HOME/.cache/huggingface" - - # HF_HUB_CACHE_MOUNT is read-only and holds the pre-staged weights below. - # WRITABLE_MODELS_DIR is writable; the benchmark script downloads anything not - # in the staged list there. - HF_HUB_CACHE_MOUNT="/scratch/models/" - WRITABLE_MODELS_DIR="/data/models/" - - # Pre-staged model - STAGED_MODELS=( - DeepSeek-R1-0528 - DeepSeek-R1-0528-NVFP4-v2 - DeepSeek-V4-Flash - DeepSeek-V4-Pro - DeepSeek-V4-Pro-0813 - GLM-5-FP8 - GLM-5-NVFP4 - GLM-5.1 - Kimi-K2.5 - Kimi-K2.5-NVFP4 - Kimi-K2.6 - Kimi-K2.6-NVFP4 - Kimi-K3 - MiniMax-M2.5 - MiniMax-M2.5-NVFP4 - MiniMax-M2.7 - MiniMax-M2.7-NVFP4 - MiniMax-M3 - MiniMax-M3-NVFP4 - Qwen3.5-397B-A17B - Qwen3.5-397B-A17B-FP8 - Qwen3.5-397B-A17B-NVFP4 - Qwen3.5-397B-A17B-NVFP4-V2 - gpt-oss-120b - ) - - # MODEL stays as the HF id for the client (--served-model-name, tokenizer); - # MODEL_PATH is what the server reads weights from. - MODEL_BASENAME="${MODEL##*/}" - if [[ $MODEL_PREFIX == "kimik2.5" && $PRECISION == "fp4" ]]; then - export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}" - elif [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then - export MODEL_PATH="${HF_HUB_CACHE_MOUNT%/}/${MODEL_BASENAME}" - else - export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}" - fi - - SQUASH_FILE="/data/squash/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh" - SPEC_SUFFIX=$([[ "$SPEC_DECODING" == "mtp" ]] && printf '_mtp' || printf '') - # Prefer a framework-tagged script (e.g. dsv4_fp4_b300_sglang.sh) so models - # with multiple inference engines can coexist; fall back to the historical - # name without an engine suffix (`_trt` for trt, bare for everyone else) - # for scripts that haven't been retagged yet. - BENCH_BASE="benchmarks/single_node/${SCENARIO_SUBDIR}${EXP_NAME%%_*}_${PRECISION}_b300" - BENCH_SCRIPT="${BENCH_BASE}_${FRAMEWORK}${SPEC_SUFFIX}.sh" - if [[ ! -f "$BENCH_SCRIPT" ]]; then - LEGACY_FW_SUFFIX=$([[ "$FRAMEWORK" == "trt" ]] && printf '_trt' || printf '') - BENCH_SCRIPT="${BENCH_BASE}${LEGACY_FW_SUFFIX}${SPEC_SUFFIX}.sh" - fi - - # Allow callers (e.g. the speedbench-al.yml AL-collection workflow) to run a - # specific script instead of the auto-selected throughput benchmark. - if [[ -n "${BENCH_SCRIPT_OVERRIDE:-}" ]]; then - BENCH_SCRIPT="$BENCH_SCRIPT_OVERRIDE" - fi - - LOCK_FILE="${SQUASH_FILE}.lock" - - # TODO(Cam): the deepseek-v4 sglang images (lmsysorg/sglang:deepseek-v4-blackwell - # and its B300-recompiled forks like yhyang201/sglang-b300) install sglang - # editable at /workspace/sglang/python (prior sglang tags used /sgl-workspace/sglang), - # so the default $GITHUB_WORKSPACE:/workspace/ bind-mount masks the install - # and breaks `import sglang`. Mount these images at /ix instead; drop the - # conditional once the image stops installing editable under /workspace. - if [[ "$IMAGE" == *deepseek-v4-blackwell* || "$IMAGE" == *deepseek-v4-bw-ultra* || "$IMAGE" == *deepseek-v4-b300* || "$IMAGE" == *sglang-b300* ]]; then - CONTAINER_MOUNT_DIR=/ix - else - CONTAINER_MOUNT_DIR=/workspace - fi - - # Import the squash file on the head node (outside any srun) under flock. - # Parallel GH jobs target the same shared squash path; flock serializes - # imports so only one job pulls and writes the file while the rest wait. - ( - exec 9>"$LOCK_FILE" - flock -w 600 9 || { echo "Failed to acquire lock for $SQUASH_FILE" >&2; exit 1; } - if unsquashfs -l "$SQUASH_FILE" > /dev/null 2>&1; then - echo "Squash file already exists and is valid, skipping import" - else - rm -f "$SQUASH_FILE" - # enroot's working dirs are pinned to NFS /scratch by - # /etc/enroot/enroot.conf, but enroot-aufs2ovlfs unpacks the image's - # root-owned whiteout markers into a sticky /tmp and then can't unlink - # them over NFS -- root-squash strips the CAP_FOWNER it would need, so - # it fails with "failed to remove aufs whiteout: Operation not - # permitted" and writes no .sqsh. Run the import on local disk, where - # the extracted files are owned by us and removable. Scoped to this - # subshell (and cleaned up on exit), so the salloc/srun below and the - # compute node's own /scratch are unaffected. - enroot_local="$(mktemp -d /tmp/enroot-import.XXXXXX)" - trap 'rm -rf "$enroot_local"' EXIT - export ENROOT_TEMP_PATH="$enroot_local/tmp" - export ENROOT_CACHE_PATH="$enroot_local/cache" - export ENROOT_DATA_PATH="$enroot_local/data" - export ENROOT_RUNTIME_PATH="$enroot_local/run" - mkdir -p "$ENROOT_TEMP_PATH" "$ENROOT_CACHE_PATH" \ - "$ENROOT_DATA_PATH" "$ENROOT_RUNTIME_PATH" - enroot import -o "$SQUASH_FILE" "docker://$IMAGE" - fi - ) - - export GPU_COUNT="${GPU_COUNT:-${TP:?TP must be set}}" - - SALLOC_ARGS=( - --partition="$SLURM_PARTITION" - --account="$SLURM_ACCOUNT" - -N 1 - --gres="gpu:$GPU_COUNT" - --exclusive - --mem=0 - --time="${SALLOC_TIME_LIMIT:-480}" - --no-shell - --job-name="$RUNNER_NAME" - ) - if [[ -n "${SALLOC_EXCLUDE:-}" ]]; then - SALLOC_ARGS+=(--exclude="$SALLOC_EXCLUDE") - fi - salloc "${SALLOC_ARGS[@]}" - JOB_ID=$(squeue --name="$RUNNER_NAME" -u "$USER" -h -o %A | head -n1) - - srun --jobid=$JOB_ID \ - --mpi=none \ - --container-image=$SQUASH_FILE \ - --container-mounts=$GITHUB_WORKSPACE:$CONTAINER_MOUNT_DIR,$HF_HUB_CACHE_MOUNT:$HF_HUB_CACHE_MOUNT,$WRITABLE_MODELS_DIR:$WRITABLE_MODELS_DIR \ - --no-container-mount-home \ - --container-remap-root \ - --container-workdir=$CONTAINER_MOUNT_DIR \ - --no-container-entrypoint --export=ALL,PORT=8888 \ - bash "$BENCH_SCRIPT" - -fi diff --git a/utils/runner_setup/RUNNER_SETUP.md b/utils/runner_setup/RUNNER_SETUP.md index 202ace0522..7fc2503edc 100644 --- a/utils/runner_setup/RUNNER_SETUP.md +++ b/utils/runner_setup/RUNNER_SETUP.md @@ -103,19 +103,19 @@ Required permissions (all of these endpoints require **admin access to the repos ``` - For example, configure 14 runners (`b300-nv_00` … `b300-nv_13`) on the B300 cluster: + For example, configure 18 runners (`b300-dsxe_00` … `b300-dsxe_17`) on the B300 DSXE cluster: ```bash ./InferenceX/utils/runner_setup/setup.sh \ AOPHAHI... \ https://github.com/actions/runner/releases/download/v2.335.1/actions-runner-linux-x64-2.335.1.tar.gz \ - 0 13 \ + 0 17 \ ~/gharunners \ - b300-nv \ + b300-dsxe \ slurm,b300 ``` - This creates `gharunner00/actions-runner` … `gharunner13/actions-runner` under the + This creates `gharunner00/actions-runner` … `gharunner17/actions-runner` under the base directory, all sharing one downloaded tarball. 5. Start the runners: @@ -134,7 +134,7 @@ Required permissions (all of these endpoints require **admin access to the repos ## Naming convention — read this before picking `BASE_RUNNER_NAME` Runner names are **load-bearing**. Each runner is named `_` -(zero-padded two-digit index), e.g. `b300-nv_07`, and two pieces of CI infrastructure +(zero-padded two-digit index), e.g. `b300-dsxe_07`, and two pieces of CI infrastructure key off that name: 1. **The launch script is selected from the name prefix.** The benchmark workflows run @@ -144,7 +144,7 @@ key off that name: ``` Everything before the first `_` must match an existing script in - [`runners/`](../../runners). For example, runner `b300-nv_07` maps to `runners/launch_b300-nv.sh`. + [`runners/`](../../runners). For example, runner `b300-dsxe_07` maps to `runners/launch_b300-dsxe.sh`. For a brand-new cluster, add a `runners/launch_.sh` first. Corollary: `BASE_RUNNER_NAME` itself must not contain `_` (use hyphens). @@ -170,7 +170,7 @@ key off that name: - Optional capacity tags, such as `b300-p1`, carve out dedicated benchmark subsets. -The per-runner name label (`b300-nv_07`) is what `runs-on` resolves for sweep jobs, so +The per-runner name label (`b300-dsxe_07`) is what `runs-on` resolves for sweep jobs, so always keep it (the script appends it automatically). A typical registered runner ends up with labels like: