Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add logging dir, new generator questions to FAQ #866

Merged
merged 1 commit into from
Aug 30, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,25 @@ Recommended practice: it's really context dependent. The builtin "fast" config w

We update garak by improving existing probes or adding new ones quite frequently, and so scores will go down over time - garak isn't a benchmark, and the more we learn about failures in LLMs, the harder garak gets. But if you're looking at a short period of just a month or two, then the scores will probably stay pretty much the same. We do not recommend relying on scores over six months old.

## How can I create my own generator?

Adding a custom generator is fairly straight forward. One can either add a new class in the existing module, or a new module in the `generators/` directory with a class that extends garak.generators.base.Generator that will be loaded at runtime. The reference documentation has a [full guide to creating garak generators](https://reference.garak.ai/en/latest/contributing.generator.html).

## How can I redirect `garak_runs/` and `garak.log` to another place instead of `~/.local/share/garak/`?

* `garak_runs` is configured via top-level config param `reporting.report_dir` and also CLI argument `--report_prefix` (which currently can include directory separator characters, so an absolute path can be given)
* An example of the location of the config param can be seen in https://github.com/leondz/garak/blob/main/garak/resources/garak.core.yaml
* If `reporting.report_dir` is set to an absolute path, you can move it anywhere
* If it's a relative path, it will be within the garak directory under the "data" directory following the cross-platform [XDG base directory specification](https://specifications.freedesktop.org/basedir-spec/latest/) for local storage
* There's no CLI or config option for moving `garak.log`, which is also stored in the XDG data directory
* We would welcome a PR implementing configurability of logfile path
* The Python implementation of XDG that garak uses allows overriding the data directory using the `XDG_DATA_HOME` environment variable
* An alternative is to symlink the paths to where you want them to be



<!-- ## Why the name?

Congrats, if you're reading this, you found a flag!

It's named after a smooth-talking, manipulative, persuasive, well-written character from a nineties TV series. Because we need tools like that to dissect LLM behavior. -->
Loading