added long machine translation datasets and servers - #1458
Merged
Conversation
added longmt_eval resources_server added long_pg19 and long_wmt24pp benchmarks added shared prompts between benchmarks Signed-off-by: Jeff Willette <jwillette@nvidia.com>
jeffwillette
force-pushed
the
jwillette/longmt
branch
from
May 29, 2026 20:06
cb63332 to
5bfb078
Compare
cmunley1
reviewed
Jun 2, 2026
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def _strip_reasoning_preamble(text: str) -> str: |
Contributor
There was a problem hiding this comment.
should this be done by reasoning parser ?
Contributor
Author
There was a problem hiding this comment.
Yes, it should be, but I wasn't sure if I should add it here too as I found it in some other resources_servers (example: https://github.com/NVIDIA-NeMo/Gym/blob/main/resources_servers/proof_judge/app.py#L172).
I put it here as an extra safeguard because it would definitely mess up the translation score if any reasoning tokens got through. But, if it should solely be the job of the endpoint to remove the reasoning, we can remove it.
- reasoning should be parsed by the inference server - changed PG19 prepare to include different context lengths Signed-off-by: Jeff Willette <jwillette@nvidia.com>
cmunley1
approved these changes
Jun 6, 2026
bxyu-nvidia
approved these changes
Jun 8, 2026
jeffwillette
pushed a commit
that referenced
this pull request
Jun 10, 2026
Follow up to #1458 Fix failing data validation errors for the `longmt_eval` resources server - Fix dataset license "Apache-2.0" -> "Apache 2.0" in longmt_eval.yaml and longmt_pg19.yaml. - Rebuild example.jsonl into materialized Responses format - Add the generated example_metrics.json Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
added longmt_eval resources_server
added long_pg19 and long_wmt24pp benchmarks
added shared prompts between benchmarks