Skip to content

fix: lower mooncake memory in functional test - #2584

Merged
yuki-97 merged 3 commits into
mainfrom
yukih/lower-mooncake-memory
May 27, 2026
Merged

fix: lower mooncake memory in functional test#2584
yuki-97 merged 3 commits into
mainfrom
yukih/lower-mooncake-memory

Conversation

@yuki-97

@yuki-97 yuki-97 commented May 27, 2026

Copy link
Copy Markdown
Contributor

Summary

fix ./tests/functional/grpo_dp_mooncake.sh fail in CI (action link) when using A100 by lower mooncake memory suggested by @ZhiyuLi-Nvidia

Test Result

confirmed in action link using 3fbc3f8.

✓ Merged data written to 
/opt/nemo-rl/tests/functional/grpo_dp_mooncake/metrics.json
                                 Metric Checks                                  
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Status ┃ Check                             ┃ Value                 ┃ Message ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ PASS   │ max(data["train/gen_kl_error"]) < │ 0.0006355919758789241 │         │
│        │ 0.002                             │                       │         │
│ PASS   │ min(data["train/probs_ratio_clam… │ 0.800000011920929     │         │
│        │ > 0.79                            │                       │         │
│ PASS   │ max(data["train/probs_ratio_clam… │ 1.0                   │         │
│        │ < 1.21                            │                       │         │
│ PASS   │ min(data["train/probs_ratio_clam… │ 1.0                   │         │
│        │ > 0.79                            │                       │         │
│ PASS   │ max(data["train/probs_ratio_clam… │ 1.2000000476837158    │         │
│        │ < 1.21                            │                       │         │
└────────┴───────────────────────────────────┴───────────────────────┴─────────┘

Signed-off-by: Yuki Huang <yukih@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented May 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yuki-97 yuki-97 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label May 27, 2026
@github-actions github-actions Bot added CI Relating to CI and removed CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) labels May 27, 2026
@yuki-97 yuki-97 added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) CI Relating to CI and removed CI Relating to CI labels May 27, 2026
@yuki-97

yuki-97 commented May 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4711c08

Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-97
yuki-97 force-pushed the yukih/lower-mooncake-memory branch from 4711c08 to 3fbc3f8 Compare May 27, 2026 09:20
@yuki-97

yuki-97 commented May 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 3fbc3f8

This reverts commit 3fbc3f8.

Signed-off-by: Yuki Huang <yukih@nvidia.com>
@github-actions github-actions Bot removed the CI Relating to CI label May 27, 2026
@yuki-97
yuki-97 marked this pull request as ready for review May 27, 2026 15:01
@yuki-97
yuki-97 requested a review from a team as a code owner May 27, 2026 15:01
@yuki-97

yuki-97 commented May 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test da9f03b

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @yuki-97 ! LGTM!

@yuki-97
yuki-97 merged commit 52d7c93 into main May 27, 2026
34 of 36 checks passed
@yuki-97
yuki-97 deleted the yukih/lower-mooncake-memory branch May 27, 2026 16:01
vigneshwaran pushed a commit to vigneshwaran/RL that referenced this pull request May 28, 2026
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants