probe: npm package hallucination

[arjun-krishna1]

• Upgrade packagehallucination probe to check for JavaScript package hallucination, looking at npm
• Implementing probe: npm package hallucination #257

[leondz]

got this:

(garak) 14:18:02 x1:~/dev/garak [feature/npm_package_hallucination] $ python -m garak -m nim -n nvidia/nemotron-mini-4b-instruct -p packagehallucination.JavaScript --parallel_requests 16 -g 3
garak LLM vulnerability scanner v0.9.0.15.post1 ( https://github.com/leondz/garak ) at 2024-08-29T14:20:30.446794
📜 logging to /home/lderczynski/.local/share/garak/garak.log
🦜 loading generator: NIM: nvidia/nemotron-mini-4b-instruct
📜 reporting to /home/lderczynski/.local/share/garak/garak_runs/garak.8e7f31b6-3504-4d4b-ad53-843d57e92277.report.jsonl
🕵️  queue of probes: packagehallucination.JavaScript
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151M/151M [00:06<00:00, 24.5MB/s]
Downloading data: 100%|█████████████████████████████████/home/lderczynski/anaconda3/envs/garak/lib/python3.12/site-packages/datasets/download/streaming_download_manager.py:778: FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.
  return pd.read_csv(xopen(filepath_or_buffer, "rb", download_config=download_config), **kwargs)
Generating train split: 3311108 examples [00:02, 1286991.22 examples/s]
Traceback (most recent call last):                                                                                                                                                     
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/lderczynski/dev/garak/garak/__main__.py", line 14, in <module>
    main()
  File "/home/lderczynski/dev/garak/garak/__main__.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/home/lderczynski/dev/garak/garak/cli.py", line 502, in main
    command.probewise_run(
  File "/home/lderczynski/dev/garak/garak/command.py", line 222, in probewise_run
    probewise_h.run(generator, probe_names, evaluator, buffs)
  File "/home/lderczynski/dev/garak/garak/harnesses/probewise.py", line 107, in run
    h.run(model, [probe], detectors, evaluator, announce_probe=False)
  File "/home/lderczynski/dev/garak/garak/harnesses/base.py", line 122, in run
    d.detect(attempt)
  File "/home/lderczynski/dev/garak/garak/detectors/packagehallucination.py", line 183, in detect
    imports = re.findall(r"import\s+(?:(?:\w+\s*,?\s*)?(?:{[^}]+})?\s*from\s+)?['\"]([^'\"]+)['\"]", o)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lderczynski/anaconda3/envs/garak/lib/python3.12/re/__init__.py", line 217, in findall
    return _compile(pattern, flags).findall(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'list'
(garak) 14:24:16 x1:~/dev/garak [feature/npm_package_hallucination] $ 

can we add a test that catches it?

[arjun-krishna1]

Hi @leondz, thanks for sharing this
I think this is an bug in the nim generator 🤔
Able to run packagehallucination.JavaScript with open ai gpt-3.5-turbo and same flags:

(.venv) arjun@arjun-ThinkPad-L14-Gen-3:~/garak$ python -m garak -m openai -n gpt-3.5-turbo -p packagehallucination.JavaScript --parallel_requests 16 -g 3
// ...
✔️  garak run complete in 211.21s

Just ran the xss probe off main with nim using the same flags as your command
And it errored as well despite being unchanged:

(.venv) arjun@arjun-ThinkPad-L14-Gen-3:~/garak$ git branch
  bugfix/litellm_provider_validation
  feature/credit_cards_probe
  feature/npm_package_hallucination
  feature/pdf_injection_probe
  feature/ruby_package_hallucination
  feature/simple_adaptive_attacks
* main
(.venv) arjun@arjun-ThinkPad-L14-Gen-3:~/garak$ python -m garak -m nim -n nvidia/nemotron-mini-4b-instruct -p xss --parallel_requests 16 -g 3
garak LLM vulnerability scanner v0.9.0.15.post1 ( https://github.com/leondz/garak ) at 2024-08-29T16:47:42.688188
📜 logging to /home/arjun/.local/share/garak/garak.log
🦜 loading generator: NIM: nvidia/nemotron-mini-4b-instruct
📜 reporting to /home/arjun/.local/share/garak/garak_runs/garak.e35ef1b6-a04c-4cbc-ad19-0654cf4fddd3.report.jsonl
🕵️  queue of probes: xss.MarkdownImageExfil
Traceback (most recent call last):                                                                                                                                                            
  File "<frozen runpy>", line 198, in _run_module_as_main                                                                                                                                     
  File "<frozen runpy>", line 88, in _run_code
  File "/home/arjun/garak/garak/__main__.py", line 14, in <module>
    main()
  File "/home/arjun/garak/garak/__main__.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/home/arjun/garak/garak/cli.py", line 509, in main
    command.probewise_run(
  File "/home/arjun/garak/garak/command.py", line 222, in probewise_run
    probewise_h.run(generator, probe_names, evaluator, buffs)
  File "/home/arjun/garak/garak/harnesses/probewise.py", line 107, in run
    h.run(model, [probe], detectors, evaluator, announce_probe=False)
  File "/home/arjun/garak/garak/harnesses/base.py", line 122, in run
    d.detect(attempt)
  File "/home/arjun/garak/garak/detectors/xss.py", line 28, in detect
    results.append(1.0 if re.search(_MARKDOWN_REGEX, output) else 0.0)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/re/__init__.py", line 177, in search
    return _compile(pattern, flags).search(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'list'

[leondz]

Thanks for the tests. I think we found a bug under parallel_requests - great!

[leondz]

refactoring suggestions - let me know what you think

[leondz]

Should this PR have been closed & locked?

[jmartin-tech]

Nope, sorry for the churn. The description in #864 triggered github to close this and attributed it to the user finalized that PR merge. ¯\(ツ)/¯

[leondz]

Oh yeah, those first words'll do it!

[leondz]

some refactoring fine-tuning. this is shaping up to be great, thanks so much for your continued contribution!

[leondz]

pr discussion unlocked

[leondz]

this looks tight!