Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

put garak data all in one place #479

Closed
leondz opened this issue Feb 15, 2024 · 6 comments · Fixed by #799
Closed

put garak data all in one place #479

leondz opened this issue Feb 15, 2024 · 6 comments · Fixed by #799
Assignees
Labels
enhancement Architectural upgrades
Milestone

Comments

@leondz
Copy link
Owner

leondz commented Feb 15, 2024

  • ~/.garak on nix/osx
  • somewhere similar on win
  • print the garak.log location near run start
  • make the location configurable
@leondz leondz added the enhancement Architectural upgrades label Feb 15, 2024
@leondz leondz self-assigned this Feb 15, 2024
@jmartin-tech
Copy link
Collaborator

Consider using the XDG specification.

@leondz leondz removed their assignment Feb 21, 2024
@leondz
Copy link
Owner Author

leondz commented Mar 24, 2024

mid-way fix: use a dir, garak_runs/ by default, for this kind of thing 660f6fd

@leondz leondz added this to the rel milestone Apr 18, 2024
@leondz
Copy link
Owner Author

leondz commented Apr 18, 2024

Some things to work out (thanks @jmartin-tech):

  • Models (ex: huggingface cache models)
  • Third party package datasets (ex: ntlk_data)
  • Sane default locations not in repo path (ex: XDG standard or user path)

@leondz leondz modified the milestones: rel, release 0.9.1 Apr 23, 2024
@leondz
Copy link
Owner Author

leondz commented May 8, 2024

XDG looks good. Makes a tonne of sense for content garak creates, unless something like a runs/ dir is specified in e.g. _config - that's probably a config node worth adding at the same time as this change

@leondz
Copy link
Owner Author

leondz commented May 8, 2024

  • visual_jailbreak download will need to comply with this

@leondz leondz changed the title put logs and runs all in one place put garak data all in one place May 13, 2024
@leondz
Copy link
Owner Author

leondz commented May 13, 2024

  • Models (ex: huggingface cache models)

Leave as default - HF uses a user-specific, not venv/conda-specific location, and data here is often large, as well as often used by other things people do

  • Third party package datasets (ex: ntlk_data)

Inclined to leave as default location. We should minimise what NLTK downloads.

  • Sane default locations not in repo path (ex: XDG standard or user path)

Let's definitely migrate to XDG default for everything directly created by garak (log, report jsonl, hitlog jsonl, report html). Possible config options: garak data dir; runs output dir (for all report & hitlog files); logfile path

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Architectural upgrades
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants