Skip to content

ci: increase Slurm time limits by +5 min for 5 nightly tests #2769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
terrykong merged 4 commits into from
Jun 11, 2026
Merged

ci: increase Slurm time limits by +5 min for 5 nightly tests #2769

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
08e5131
ci: increase Slurm time limits by 10% for 5 nightly tests
kajalj22 Jun 10, 2026
4b19a89
fixup! ci: increase Slurm time limits by 10% for 5 nightly tests
kajalj22 Jun 10, 2026
2c9cbd6
Merge remote-tracking branch 'origin/main' into increase-slurm-time-l…
kajalj22 Jun 10, 2026
fe2a945
Merge branch 'main' into increase-slurm-time-limits
kajalj22 Jun 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tests/test_suites/llm/grpo-nanov3-30BA3B-2n8g-fsdp2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ GPUS_PER_NODE=8
STEPS_PER_RUN=10
MAX_STEPS=10
NUM_RUNS=$(( (MAX_STEPS + STEPS_PER_RUN - 1) / STEPS_PER_RUN )) # Round up
NUM_MINUTES=45
NUM_MINUTES=50
# ===== END CONFIG =====

exit_if_max_steps_reached
Expand Down
2 changes: 1 addition & 1 deletion tests/test_suites/llm/grpo-nanov3-30BA3B-2n8g-megatron-pack-cp-tq_simple.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ GPUS_PER_NODE=8
STEPS_PER_RUN=3
MAX_STEPS=3
NUM_RUNS=$(( (MAX_STEPS + STEPS_PER_RUN - 1) / STEPS_PER_RUN )) # Round up
NUM_MINUTES=30
NUM_MINUTES=35
# ===== END CONFIG =====

source "$SCRIPT_DIR/common-tq.env"
Expand Down
2 changes: 1 addition & 1 deletion tests/test_suites/llm/grpo-qwen2.5-32b-32n8g-fsdp2tp8-actckpt.v3.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ NUM_NODES=32
STEPS_PER_RUN=2 # step_time: [220, 310]
MAX_STEPS=2
NUM_RUNS=$(( (MAX_STEPS + STEPS_PER_RUN - 1) / STEPS_PER_RUN )) # Round up
NUM_MINUTES=30
NUM_MINUTES=35
# ===== END CONFIG =====

exit_if_max_steps_reached
Expand Down
2 changes: 1 addition & 1 deletion tests/test_suites/llm/sft-nanov3-30BA3B-2n8g-fsdp2-lora.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ NUM_NODES=2
STEPS_PER_RUN=20 # step_time ~ 10sec
MAX_STEPS=20
NUM_RUNS=$(( (MAX_STEPS + STEPS_PER_RUN - 1) / STEPS_PER_RUN )) # Round up
NUM_MINUTES=30 # Usually 15 minutes is enough for 20 steps, but we add a buffer of 3 minutes in metrics check (30min to buffer for initial ckpt download)
NUM_MINUTES=35 # Usually 15 minutes is enough for 20 steps, but we add a buffer of 3 minutes in metrics check (30min to buffer for initial ckpt download)
# ===== END CONFIG =====

exit_if_max_steps_reached
Expand Down
2 changes: 1 addition & 1 deletion tests/test_suites/llm/sft-qwen2.5-32b-4n8g-fsdp2tp8sp-actckpt.v3.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ NUM_NODES=4
STEPS_PER_RUN=20 # step_time ~ 29sec
MAX_STEPS=20
NUM_RUNS=$(( (MAX_STEPS + STEPS_PER_RUN - 1) / STEPS_PER_RUN )) # Round up
NUM_MINUTES=30
NUM_MINUTES=35
# ===== END CONFIG =====

exit_if_max_steps_reached
Expand Down
Loading