Skip to content

Releases: leondz/garak

v0.9.0.16

06 Sep 15:28
5f473f1
Compare
Choose a tag to compare

What's Changed

New plugins

New features

  • extract generations param to be set on probes instead of generators by @jmartin-tech in #837
  • secure garak HF assets by @leondz in #854
  • Rename knownbadsignatures to av_spam_scanning by @leondz in #850
  • enable tokenizer customization in HFDetector by @jmartin-tech in #855
  • refactor calibration / z-score code, so z-scores can be shown in CLI at run time by @leondz in #847

Tuning & fixes

New Contributors

Full Changelog: v0.9.0.15...v0.9.0.0.16

v0.9.0.15

13 Aug 15:51
Compare
Choose a tag to compare

What's Changed

New plugins

  • probe: assess πŸ€— repos for potentially malicious files (fileformats) by @leondz in #767

New features

  • use XDG paths for configuration data and caching by @jmartin-tech in #799
  • Report comparative info for detector scores by @leondz in #814
  • enable HFDetector model configuration with hf_args by @jmartin-tech in #810
  • Flags for varying NIM seed and temperature every call by @leondz in #808

Tuning & fixes

  • don't skip listing probes in 100%-pass-rate modules by @leondz in #781
  • make default regex raw to remove python 3.12 warnings by @jmartin-tech in #809
  • suppress HF warnings and lower during detector model instantiation by @leondz in #807
  • validate cache consistency on first access by @jmartin-tech in #815
  • clarify continuation probes by @leondz in #813
  • print logfile path when invoked from CLI by @leondz in #811
  • update slurprompts filename in encoding pkg by @leondz in #818

Full Changelog: v0.9.0.14...v0.9.0.15

v0.9.0.14

29 Jul 13:24
Compare
Choose a tag to compare

What's Changed

New plugins

New features

  • Feature: configurable plugins by @jmartin-tech in #711
  • Add JSON Path support for RestGenerator by @leondz in #651
  • Make banner line test dynamic by @jmartin-tech in #670
  • Tutorial for contributing a generator by @leondz in #688
  • add 'complete' message to report at end of run by @leondz in #735
  • support chat modality in attempts by @leondz in #644
  • add grandma substance synth probe, update docs and var names by @leondz in #763
  • update NVCF to support versioning, custom payload keys by @leondz in #772
  • WINDOWS & OSX SUPPORT: test by default on linux, osx, win by @leondz in #793
  • enable nested setting of plugin configs by @leondz in #773
  • always log nvcf payload by @leondz in #778
  • update ecoji for windows support by @jmartin-tech in #787
  • remove extraneous default params for nims that expect conservative pa… by @leondz in #749
  • require probes to match input modality of the generator by @jmartin-tech in #738
  • Improve cloze data for leakreplay probe by @leondz in #708
  • add CONTRIBUTING.md, update contributing guide in docs by @leondz in #706
  • add DEFAULT_PARAMS to base types that are configurables by @leondz in #771
  • atkgen - add configurability, allow config of which adversarial generator to use by @leondz in #777
  • enhancement: enable lazy setting of nested dicts by @leondz in #775
  • convert mutation_generator to use load_plugin by @jmartin-tech in #795
  • Add support for gpt-4o, gpt-4o-mini by @erickgalinkin in #798
  • bump transformers ver to min reqd for llama 3.1 by @leondz in #800
  • add mitigation strings, add test for string-based mitigation skipping by @leondz in #762
  • speedup report generation by using class attribs instead of instantiating by @leondz in #707
  • cached plugin enum by @jmartin-tech in #768

Tuning & fixes

Full Changelog: v0.9.0.13...v0.9.0.14

v0.9.0.13

10 May 13:41
Compare
Choose a tag to compare

What's Changed

garak's 1st birthday! πŸŽ‚

Headlines in this birthday release:

  • Multimodal support! LLaVa + FigStep - HUGE thanks to @DavidLee528
  • LiteLLM generator support - thanks to @Tien-Cheng!
  • DoNotAnswer probe for prompts an LLM should decline - thanks to @AhsanAyub!
  • LangChain Serve generator support - thanks to @GustavFredrikson!
  • Support for NIM generators, NVIDIA Inference Microservices
  • OpenAI abstraction with parallelisation support
  • Windows compatibility enhanced, with optional testing in the workflow
  • Hugging Face speedup

plugins

architecture

stability

New Contributors

Full Changelog: v0.9.0.12...v0.9.0.13

v0.9.0.12

20 Feb 16:12
Compare
Choose a tag to compare

What's Changed

plugins

functionality improvements

  • support multiple buffs by @leondz in #497
  • wrap exception printing in repr by @leondz in #425
  • add generators.function docs & examples by @leondz in #437
  • update doc indices, add test to check them by @leondz in #450
  • fix & unify REST generator timeout param names; set default request timeout to 20s by @leondz in #451
  • add test to keep requirements in sync by @leondz in #465
  • docs for buffs by @leondz in #466
  • autosearch in the configs/ subdir for configs (no yaml extension should be given) by @leondz in #467
  • Update function.py by @erickgalinkin in #500
  • add warning when using a lite/default profile by @leondz in #476
  • rename default output dir to garak_runs/; by @leondz in #488
  • update openai model list by @leondz in #494
  • make test_openai generation tests skip if no OAI API key set by @leondz in #491

fixes

New Contributors

Full Changelog: v0.9.0.11...v0.9.0.12

v0.9.0.11.post1

26 Jan 14:02
e41db8e
Compare
Choose a tag to compare

What's Changed

Full Changelog: v0.9.0.11...v0.9.0.11.post1

v0.9.0.11

19 Jan 09:52
Compare
Choose a tag to compare

What's Changed

  • Probe for repetition-based nudging into replay/spurious generation by @leondz in #404
  • Probe for invisible text prompt injections by @leondz in #397
  • Probe for the 'DAN in the wild' paper's library of jailbreak prompts by @leondz in #405
  • Probe for NYT & The Guardian content in training data by @leondz in #402
  • Add NVIDIA cloud functions generator by @leondz in #398
  • Add toxicity generation deep test config by @leondz in #413
  • Generator enhancements and minor improvements by @shubhobm in #391
  • Update HF inf api generator to match their current expectations by @leondz in #400
  • Invoke garak on the command line, with garak by @leondz in #410
  • Mitigate continuation probe oversensitivity by @leondz in #394
  • Handle nvcf container timeouts by @leondz in #399
  • Fixing Exception Cause By Type Error When Scanning LLMs Via Replicate by @DavidLee528 in #401
  • Make sure triggers attempt.note is saved in hitlog by @leondz in #403
  • Repeat replay now optionally overrides generator max len by @leondz in #408
  • Replay.Repeat now preserves attempt when restoring generator max_tokens by @leondz in #409
  • Gracefully handle NVCF request timeouts & other failures by @leondz in #411
  • Fix deprecated encoding by @leondz in #412
  • Better coverage in mitigation bypass detector

Full Changelog: v0.9.0.10...v0.9.0.11

v0.9.0.10

20 Dec 13:35
Compare
Choose a tag to compare
  • Probes can now be selected by MISP tag, e.g. owasp:llm01
  • garak now automatically creates an HTML report on completion
  • HTML reports can be grouped by module but also by probe tag category, so you can see e.g. top-level scores for prompt injection, hallucination, and so on
  • logs now go to a dedicated log dir by default, to keep things clean
  • new buffs: encoding.Base64, encoding.CharCode
  • new generator: NeMo guardrails
  • new probe: AutoDAN
  • RealToxicityProbes now only loads local lists, much faster
  • update OpenAI models list
  • fix attempt parameter stability
  • better logging of config params
  • atk is now atkgen

Contributions from @erickgalinkin , @drazvan . Enjoy & Happy holidays! πŸŽ…πŸŽ„

What's Changed

New Contributors

Full Changelog: v0.9.0.9...v0.9.0.10

v0.9.0.9

08 Dec 13:40
Compare
Choose a tag to compare

garak v0.9.0.9

  • Added GCG jailbreak probe (probes.gcg.GCG_Probe)
  • Add support for NVIDIA Optimum (generators.huggingface.OptimumPipeline)
  • Add OWASP tags to probes
  • Add fast & slow paraphrase buffs (buffs.paraphrase.Fast, buffs.paraphrase.PegasusT5)
  • Support for config files: there's a core config, site config, and a CLI config, and all can be used to set system, run, and plugin parameters
  • Supply some sample config files for a few different styles of garak run
  • Progress bar for buffs
  • Added debugging REST server for dev
  • Move RealToxicityPrompts resources to their own subdir

Thanks to @erickgalinkin @drazvan @DavidLee528

v0.9.0.8

14 Nov 16:17
Compare
Choose a tag to compare
  • Rename ART to AG (Attack Generator)
  • Add generator support for NeMo LLM
  • Add generator support for OctoML
  • Add generic REST connector, with configs
  • Add option to parallelise requests
  • Add option to parallelise attempts
  • Include AutoDAN probe
  • Added "interactive mode", where you get a garak CLI πŸŽ‰
  • Fix continuation probe trigger alignment
  • Fix RTP prompts to be aggressive
  • Add support for langchain LLM interface
  • Upgrade in avidtools
  • Improve checking for detector names in probes
  • Turn-by-turn visual indicator on attack generator probe