Skip to content

feat: vllm Model diagnostic test checking long generation quality#516

Merged
SahilJain314 merged 5 commits intoNVIDIA-NeMo:mainfrom
vegaluisjose:patch-1
Jun 27, 2025
Merged

feat: vllm Model diagnostic test checking long generation quality#516
SahilJain314 merged 5 commits intoNVIDIA-NeMo:mainfrom
vegaluisjose:patch-1

Conversation

@vegaluisjose
Copy link
Contributor

@vegaluisjose vegaluisjose commented Jun 13, 2025

What does this PR do ?

Model diagnostic test checking long generation quality @terrykong

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

@vegaluisjose
Create 2.long_generation_decode_vs_prefill.py test
857eb02 
Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
terrykong
terrykong reviewed Jun 13, 2025
View reviewed changes
Copy link
Collaborator

@terrykong terrykong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vegaluisjose !

Could you also add an entry below this one for this diagnostic script and briefly explain what the script tests?

terrykong
terrykong reviewed Jun 13, 2025
View reviewed changes
@terrykong terrykong changed the title Model diagnostic test checking long generation quality feat: vllm Model diagnostic test checking long generation quality Jun 16, 2025
@vegaluisjose
Change threshold value
e40b1d3 
Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
@vegaluisjose
Update adding-new-models.md
bc99af4 
Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Jun 16, 2025
@vegaluisjose
Copy link
Contributor Author

vegaluisjose commented Jun 16, 2025

Updated docs with test here, https://github.com/vegaluisjose/RL/blob/patch-1/docs/adding-new-models.md#2long_generation_decode_vs_prefill

@terrykong let me know what you think

terrykong
terrykong reviewed Jun 16, 2025
View reviewed changes
@terrykong
Update docs/adding-new-models.md
92a4e05 
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
terrykong
terrykong previously approved these changes Jun 16, 2025
View reviewed changes
@terrykong terrykong enabled auto-merge June 16, 2025 17:58
parthchadha
parthchadha previously approved these changes Jun 17, 2025
View reviewed changes
@terrykong terrykong added this pull request to the merge queue Jun 18, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 19, 2025
@vegaluisjose
fixed formatting in test
d830723 
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
@vegaluisjose vegaluisjose dismissed stale reviews from parthchadha and terrykong via d830723 June 26, 2025 22:45
@SahilJain314 SahilJain314 added this pull request to the merge queue Jun 26, 2025
Merged via the queue into NVIDIA-NeMo:main with commit 2e339dd Jun 27, 2025
13 of 14 checks passed
xxman-google pushed a commit to xxman-google/NeMo-RL that referenced this pull request Jun 27, 2025
@vegaluisjose @terrykong @xxman-google
feat: vllm Model diagnostic test checking long generation quality (NV…
d3210b2 
…IDIA-NeMo#516)

Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Terry Kong <terrycurtiskong@gmail.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
xxman-google pushed a commit to xxman-google/NeMo-RL that referenced this pull request Jun 30, 2025
@vegaluisjose @terrykong @xxman-google
feat: vllm Model diagnostic test checking long generation quality (NV…
fb6c526 
…IDIA-NeMo#516)

Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Terry Kong <terrycurtiskong@gmail.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Signed-off-by: Xuehan <xxman@google.com>
xxman-google pushed a commit to xxman-google/NeMo-RL that referenced this pull request Jun 30, 2025
@vegaluisjose @terrykong @xxman-google
feat: vllm Model diagnostic test checking long generation quality (NV…
f431d48 
…IDIA-NeMo#516)

Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Terry Kong <terrycurtiskong@gmail.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Signed-off-by: Xuehan <xxman@google.com>
therealnaveenkamal pushed a commit to therealnaveenkamal/RL that referenced this pull request Jul 7, 2025
@vegaluisjose @terrykong @therealnaveenkamal
feat: vllm Model diagnostic test checking long generation quality (NV…
013c848 
…IDIA-NeMo#516)

Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Terry Kong <terrycurtiskong@gmail.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
YzjiaoNvd pushed a commit to YzjiaoNvd/NeMo-RL that referenced this pull request Jul 14, 2025
@vegaluisjose @terrykong
feat: vllm Model diagnostic test checking long generation quality (NV…
bbb17dc 
…IDIA-NeMo#516)

Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Terry Kong <terrycurtiskong@gmail.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
KiddoZhu pushed a commit that referenced this pull request Jul 28, 2025
@vegaluisjose @terrykong @KiddoZhu
feat: vllm Model diagnostic test checking long generation quality (#516)
d2b7ae5 
Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Terry Kong <terrycurtiskong@gmail.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
FannYYW pushed a commit to xxman-google/NeMo-RL that referenced this pull request Aug 5, 2025
@vegaluisjose @terrykong @xxman-google
feat: vllm Model diagnostic test checking long generation quality (NV…
4675f1b 
…IDIA-NeMo#516)

Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Terry Kong <terrycurtiskong@gmail.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@terrykong terrykong terrykong left review comments

+2 more reviewers

@SahilJain314 SahilJain314 SahilJain314 approved these changes

@parthchadha parthchadha parthchadha left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vegaluisjose @terrykong @parthchadha @SahilJain314