fix: flaky fireworks-e2e test - #2753
Merged
Merged
Conversation
The fireworks-e2e test intermittently fails verify_inference_provider_rollout.py's assertion that the model emits exactly one function_call, because sampling was unconstrained and the model would sometimes answer in plain text instead of calling get_weather. gpu_e2e_test.sh already pins --temperature 0 for the same determinism reason; apply the same fix here. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Contributor
Author
|
/ok to test ede63c8 |
--temperature 0 did not fix the fireworks-e2e flake (still failed with the same empty function_calls assertion), so this isn't sampling-driven flakiness. Upload rollouts.jsonl/gym.log from provider_e2e_tests as artifacts, mirroring the existing gpu_e2e_tests pattern, so the next failure can be inspected to find the real root cause instead of guessing. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
kajalj22
changed the title
chtruong814
previously approved these changes
Aug 25, 2026
Contributor
Author
|
/ok to test a5691df |
Root cause found via the artifact uploaded in the previous commit: the test
reused the shared example_single_tool_call dataset's first row ("what's it
like in sf?"), which is ambiguous enough that gpt-oss-20b sometimes answers
directly in text instead of calling get_weather. Neither temperature=0 nor
the parsing path was at fault -- the raw response genuinely had no tool call.
Add tests/e2e/inference_provider_smoke.jsonl, a dedicated e2e prompt with an
unambiguous weather request and tool_choice=required, mirroring the existing
tests/e2e/gpu_smoke.jsonl pattern (gpu_e2e_test.sh also uses its own smoke
file rather than a shared dataset). Point run_inference_provider_e2e.sh at it.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Contributor
Author
|
/ok to test b725ba1 |
kajalj22
changed the title
test_provider_e2e_matrix_selects_config_model_and_secret_by_name asserted the old shared example.jsonl path was present in run_inference_provider_e2e.sh. Update it to check for the new dedicated inference_provider_smoke.jsonl path. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Contributor
Author
|
/ok to test 5ad0411 |
2 tasks
tool_choice=required forces a tool call on every agent step. With max_steps=2, after the model correctly calls get_weather on step 1, step 2 is forced to call a tool again even though there's nothing left to ask, so it repeats the identical get_weather call -- producing 2 function_call items instead of 1 and failing the verify script's exact-one-call assertion. The unambiguous prompt plus --temperature 0 already produces exactly one correct tool call without forcing it, so drop tool_choice entirely. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Contributor
Author
|
/ok to test 5512587 |
kajalj22
changed the title
chtruong814
approved these changes
Aug 25, 2026
Contributor
Author
|
/ok to test f360125 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/e2e/run_inference_provider_e2e.shnow runs against a dedicatedtests/e2e/inference_provider_smoke.jsonlprompt (unambiguous weather request) instead of the sharedexample_single_tool_calldataset, and pins--temperature 0for determinism. Mirrors the existingtests/e2e/gpu_smoke.jsonlpattern used bygpu_e2e_test.sh.provider_e2e_testsincicd-main.ymlnow uploadsrollouts.jsonl/gym.logas a build artifact on every run (mirrorsgpu_e2e_tests), for debugging future failures.Test plan
fireworks-e2eCI job passes on this PR