Skip to content

docs: Add missing arguments to DeepScaler evaluation#502

Merged
SahilJain314 merged 2 commits intoNVIDIA-NeMo:mainfrom
butsugiri:fix-deepscaler-arguments
Jun 30, 2025
Merged

docs: Add missing arguments to DeepScaler evaluation#502
SahilJain314 merged 2 commits intoNVIDIA-NeMo:mainfrom
butsugiri:fix-deepscaler-arguments

Conversation

@butsugiri
Copy link
Contributor

@butsugiri butsugiri commented Jun 11, 2025

What does this PR do ?

This PR attempts to fix the document for deepscaler experiments

Currently, necessary arguments are missing, which leads to poor evaluation results

============================================================
model_name='step_300-hf' dataset_name='aime_2024'
max_new_tokens=2048 temperature=0.0 top_p=1.0 top_k=-1

metric='pass@1' num_tests_per_prompt=1

score=0.0333 (1.0/30)
============================================================

By specifying cot.txt (as is done in training setup), the result improves a bit.

============================================================
model_name='step_300-hf' dataset_name='aime_2024'
max_new_tokens=2048 temperature=0.0 top_p=1.0 top_k=-1

metric='pass@1' num_tests_per_prompt=1

score=0.1333 (4.0/30)
============================================================

By allowing the generation of more than 2048 tokens, the result is even better (This PR)

============================================================
model_name='step_300-hf' dataset_name='aime_2024'
max_new_tokens=8192 temperature=0.0 top_p=1.0 top_k=-1

metric='pass@1' num_tests_per_prompt=1

score=0.3667 (11.0/30)
============================================================

Issues

n/a

Usage

n/a

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests? --> n/a
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests --> n/a
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs. --> n/a

Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Jun 11, 2025
@butsugiri butsugiri changed the title Add missing arguments to DeepScaler evaluation docs: Add missing arguments to DeepScaler evaluation Jun 11, 2025
@parthchadha parthchadha requested a review from abukharin-nv June 26, 2025 22:19
@SahilJain314
Copy link
Contributor

Thanks for the PR! Slipped past us for a bit.

abukharin-nv
abukharin-nv previously approved these changes Jun 27, 2025
Copy link
Contributor

@abukharin-nv abukharin-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I would also suggest increasing max_len to 32K, but that is kind of a subjective choice.

@butsugiri
Copy link
Contributor Author

@abukharin-nv
Thank you for reviewing my PR! I have set max_len to 32768, and it gave me the following results:

--> I am updating my PR.

============================================================
model_name='step_300-hf' dataset_name='aime_2024'
max_new_tokens=32768 temperature=0.0 top_p=1.0 top_k=-1

metric='pass@1' num_tests_per_prompt=1

score=0.3667 (11.0/30)
============================================================

Please let me know if there's anything I can improve.

Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>
@butsugiri butsugiri force-pushed the fix-deepscaler-arguments branch from 3e00e33 to d1564af Compare June 30, 2025 01:35
@SahilJain314 SahilJain314 enabled auto-merge June 30, 2025 04:10
@SahilJain314 SahilJain314 added this pull request to the merge queue Jun 30, 2025
Merged via the queue into NVIDIA-NeMo:main with commit 0b5550f Jun 30, 2025
12 of 14 checks passed
xxman-google pushed a commit to xxman-google/NeMo-RL that referenced this pull request Jun 30, 2025
Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>
xxman-google pushed a commit to xxman-google/NeMo-RL that referenced this pull request Jul 2, 2025
Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>
therealnaveenkamal pushed a commit to therealnaveenkamal/RL that referenced this pull request Jul 7, 2025
Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>
YzjiaoNvd pushed a commit to YzjiaoNvd/NeMo-RL that referenced this pull request Jul 14, 2025
Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>
KiddoZhu pushed a commit that referenced this pull request Jul 28, 2025
Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>
FannYYW pushed a commit to xxman-google/NeMo-RL that referenced this pull request Aug 5, 2025
Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants