Skip to content

ci: Temp disable megatron lora grpo tests #2062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chtruong814 wants to merge 1 commit into from
Closed

ci: Temp disable megatron lora grpo tests #2062

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions tests/functional/L1_Functional_Tests_GPU.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ run_test uv run --no-sync bash ./tests/functional/grpo_automodel_lora_async
run_test uv run --no-sync bash ./tests/functional/grpo_automodel_lora_non_colocated.sh
run_test uv run --no-sync bash ./tests/functional/grpo_megatron.sh
run_test uv run --no-sync bash ./tests/functional/grpo_megatron_generation.sh
run_test uv run --no-sync bash ./tests/functional/grpo_megatron_lora.sh
run_test uv run --no-sync bash ./tests/functional/grpo_megatron_lora_async.sh
# run_test uv run --no-sync bash ./tests/functional/grpo_megatron_lora.sh
# run_test uv run --no-sync bash ./tests/functional/grpo_megatron_lora_async.sh
Comment on lines +55 to +56

@coderabbitai coderabbitai Bot Mar 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid silently removing unique CI coverage; gate these tests with an explicit temporary switch.

Line 55 and Line 56 currently remove the only L1 coverage for GRPO+Megatron+LoRA (including async), which means these paths won’t run in the main CI flow (.github/workflows/cicd-main.yml:310-327, plus main/scheduled L1 behavior in lines 100-120 of that workflow). Please keep these invocations in place behind an explicit skip flag and log the tracking PR/issue so re-enable criteria stays visible.

Proposed change 
-# run_test      uv run --no-sync bash ./tests/functional/grpo_megatron_lora.sh
-# run_test      uv run --no-sync bash ./tests/functional/grpo_megatron_lora_async.sh
+if [[ "${DISABLE_GRPO_MEGATRON_LORA_TESTS:-1}" == "1" ]]; then
+    echo "Temporarily skipping GRPO Megatron LoRA tests (tracking fix: PR `#1889`)"
+else
+    run_test      uv run --no-sync bash ./tests/functional/grpo_megatron_lora.sh
+    run_test      uv run --no-sync bash ./tests/functional/grpo_megatron_lora_async.sh
+fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# run_test uv run --no-sync bash ./tests/functional/grpo_megatron_lora.sh
# run_test uv run --no-sync bash ./tests/functional/grpo_megatron_lora_async.sh
if [[ "${DISABLE_GRPO_MEGATRON_LORA_TESTS:-1}" == "1" ]]; then
echo "Temporarily skipping GRPO Megatron LoRA tests (tracking fix: PR `#1889`)"
else
run_test uv run --no-sync bash ./tests/functional/grpo_megatron_lora.sh
run_test uv run --no-sync bash ./tests/functional/grpo_megatron_lora_async.sh
fi
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@tests/functional/L1_Functional_Tests_GPU.sh` around lines 55 - 56, Restore
the two commented-out run_test lines for
./tests/functional/grpo_megatron_lora.sh and
./tests/functional/grpo_megatron_lora_async.sh and wrap them with an explicit
gate (e.g., check SKIP_L1_GRPO_MEGATRON_LORA environment variable or a
--skip-l1-grpo flag) so the tests remain in CI but can be skipped; when
skipping, emit a clear log message that includes the tracking PR/issue
identifier (e.g., "Skipping L1 GRPO+Megatron+LoRA tests per PR#1234/issue#5678")
so re-enable criteria are visible. Ensure the gate logic is placed where the
original run_test invocations were removed and that the skip variable defaults
to false so coverage runs unless explicitly set.

run_test uv run --no-sync bash ./tests/functional/grpo_multiple_dataloaders.sh
run_test uv run --no-sync bash ./tests/functional/grpo_multiturn.sh
run_test uv run --no-sync bash ./tests/functional/grpo_non_colocated.sh
Expand Down
Loading