Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove extraneous default params for nims that expect conservative pa… #749

Merged
merged 11 commits into from
Jun 25, 2024

Conversation

leondz
Copy link
Collaborator

@leondz leondz commented Jun 19, 2024

resolves #748

  • add a "blocked params" value to OpenAICompatible so we can control verbosity of payloads
  • strip out Nones from OpenAI API payloads
  • force unset defaults for some params in nim

one question - why are these defaults defined twice?

    # template defaults optionally override when extending
    DEFAULT_PARAMS = Generator.DEFAULT_PARAMS | {
        "temperature": 0.7,
        "top_p": 1.0,
        "frequency_penalty": 0.0,
        "presence_penalty": 0.0,
        "stop": ["#", ";"],
        "blocked_params": set(),
    }

    temperature = 0.7
    top_p = 1.0
    frequency_penalty = 0.0
    presence_penalty = 0.0
    stop = ["#", ";"]
    blocked_params = set()

…yloads; add ability to block payload params by name
@leondz leondz added bug Something isn't working generators Interfaces with LLMs labels Jun 19, 2024
@leondz leondz requested a review from jmartin-tech June 19, 2024 12:38
@leondz
Copy link
Collaborator Author

leondz commented Jun 19, 2024

-- ideally would have liked to have done this via config, not sure it woulda been possible though

Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIce catch, I see a few more places we could improve code reuse.

garak/generators/openai.py Outdated Show resolved Hide resolved
garak/generators/openai.py Outdated Show resolved Hide resolved
garak/generators/openai.py Outdated Show resolved Hide resolved
garak/generators/nim.py Outdated Show resolved Hide resolved
@leondz leondz requested a review from jmartin-tech June 25, 2024 06:07
Copy link
Collaborator

@erickgalinkin erickgalinkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Minor comment and suggestion.

@@ -38,6 +38,7 @@ class NVOpenAIChat(OpenAICompatible):
"top_p": 0.7,
"top_k": 0, # top_k is hard set to zero as of 24.04.30
"uri": "https://integrate.api.nvidia.com/v1/",
"suppressed_params": {"n", "frequency_penalty", "presence_penalty"},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a ref for these we can throw into a comment so we can keep it up to date?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was learned through trial by 422 / 400 error at the moment

garak/generators/openai.py Outdated Show resolved Hide resolved
Co-authored-by: Erick Galinkin <[email protected]>
Signed-off-by: Leon Derczynski <[email protected]>
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I would like to see the new NIM test here updated to use the openai_compat_mocks fixture. Happy to make that a follow on I will PR shortly.

@jmartin-tech jmartin-tech merged commit d49fab0 into main Jun 25, 2024
6 checks passed
@jmartin-tech jmartin-tech deleted the bugfix/nim422 branch June 25, 2024 21:22
@github-actions github-actions bot locked and limited conversation to collaborators Jun 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working generators Interfaces with LLMs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

nim openai endpoints reject requests with unexpected params in payload
3 participants