Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Guard cli run with invalid or incomplete config #677

Merged
merged 6 commits into from
May 16, 2024
Merged

Guard cli run with invalid or incomplete config #677

merged 6 commits into from
May 16, 2024

Conversation

jmartin-tech
Copy link
Collaborator

Fix #191

  • probe/detector/buff configuration checks
    • invalid or missing values generate an error message and exit early
    • a [bool] cli argument --skip_unknown logs unknown configurations & continues to start_run
  • front load argument parsing for probes and detectors
    • defers run log creation until run config is validated

Local test examples

python -m garak -m test -n mistralai/mixtral-8x22b-v0.1 -g 1 -p dan.DanInTheWild,doesnotexist,dan.Invalid --skip_unknown
python -m garak -m test -n mistralai/mixtral-8x22b-v0.1 -g 1 -p dan.DanInTheWild,doesnotexist
python -m garak -m test -n mistralai/mixtral-8x22b-v0.1 -g 1 -p doesnotexist
python -m garak -m test -n mistralai/mixtral-8x22b-v0.1 -g 1 -p doesnotexist --skip_unknown

@jmartin-tech
Guard cli run with invalid or incomplete config
6f18039 
Signed-off-by: Jeffrey Martin <[email protected]>
@jmartin-tech
change subclass spec selection
8eb068d 
* enforce full class namespace for sub selection
* add testing documentation of behavior change

Signed-off-by: Jeffrey Martin <[email protected]>
@jmartin-tech
print skipped arg requested spec items
ea87654 
* docuement that full class overrides default state
* log and print skipped spec items

Signed-off-by: Jeffrey Martin <[email protected]>
@jmartin-tech
update parse_plugin_spec return type & cli test
660fe95 
* return type for parse_plugin_spec is not a tuple
* cli testing with `invalid` probe names needs `--skip_unknown` arg

Signed-off-by: Jeffrey Martin <[email protected]>
leondz
leondz reviewed May 13, 2024
View reviewed changes
Copy link
Collaborator

@leondz leondz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Good tests. It'll be nice to refactor cli.py, and it looks tidier now. Some minor questions below but I think just one change that is potentially at "required" level for this PR.

garak/cli.py Outdated Show resolved Hide resolved
garak/_config.py Show resolved Hide resolved
garak/_config.py Show resolved Hide resolved
garak/cli.py Outdated Show resolved Hide resolved
garak/cli.py Show resolved Hide resolved
@jmartin-tech
make skip_unknown a system_param
b7ed214 
Signed-off-by: Jeffrey Martin <[email protected]>
@leondz
Copy link
Collaborator

leondz commented May 14, 2024

after b7ed214, lgtm

@leondz leondz merged commit 324f638 into NVIDIA:main May 16, 2024
4 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators May 16, 2024
@jmartin-tech jmartin-tech deleted the feature/exit-when-no-work branch May 16, 2024 14:16
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@leondz leondz leondz left review comments

@erickgalinkin erickgalinkin Awaiting requested review from erickgalinkin

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

fail if non-existing plugin is requested
2 participants
@jmartin-tech @leondz